Problems with Strings

I am accessing IRIS databases with JDBC (or ODBC) using Python. I want to fetch the data into a pandas dataframe to manipulate the data and create charts from it. I ran into a problem with string handling while using JDBC. This post is to help if anyone else has the same issues. Or, if there is an easier way to solve this, let me know in the comments!

I am using OSX, so I am unsure how unique my problem is. I am using Jupyter Notebooks, although the code would generally be the same if you used any other Python program or framework.

The JDBC problem

When I fetch data from the database the column descriptions and any string data are returned as data type java.lang.String. If you print string data data it will look like: "(p,a,i,n,i,n,t,h,e,r,e,a,r)" instead of the expected "painintherear".

This is probably because character strings of data type java.lang.String are coming through as an iterable or array when fetched using JDBC. This can happen if the Python-Java bridge you're using (e.g., JayDeBeApi, JDBC) is not automatically converting java.lang.String to a Python str in a single step.

Python's str string representation, in contrast, has the whole string as a single unit. When Python retrieves a normal str (e.g. via ODBC), it doesn't split into individual characters.

The JDBC Solution

To fix this issue, you must ensure that the java.lang.String is correctly converted into Python's str type. You can explicitly handle this conversion when processing the fetched data so it is not interpreted as an iterable or list of characters.

There are many ways to do this string manipulation; this is what I did.

import pandas as pd

import pyodbc

import jaydebeapi
import jpype

def my_function(jdbc_used)

    # Some other code to create the connection goes here

    cursor.execute(query_string)

    if jdbc_used:
        # Fetch the results, convert java.lang.String in the data to Python str
        # (java.lang.String is returned "(p,a,i,n,i,n,t,h,e,r,e,a,r)" Convert to str type "painintherear"
        results = []
        for row in cursor.fetchall():
            converted_row = [str(item) if isinstance(item, jpype.java.lang.String) else item for item in row]
            results.append(converted_row)

        # Get the column names and ensure they are Python strings 
        column_names = [str(col[0]) for col in cursor.description]

        # Create the dataframe
        df = pd.DataFrame.from_records(results, columns=column_names)

        # Check the results
        print(df.head().to_string())

    else:  
        # I was also testing ODBC
        # For very large result sets get results in chunks using cursor.fetchmany(). or fetchall()
        results = cursor.fetchall()
        # Get the column names
        column_names = [column[0] for column in cursor.description]
        # Create the dataframe
        df = pd.DataFrame.from_records(results, columns=column_names)

    # Do stuff with your dataframe

The ODBC problem

When using an ODBC connection, strings are not returned or are NA.

If you're connecting to a database that contains Unicode data (e.g., names in different languages) or if your application needs to store or retrieve non-ASCII characters, you must ensure that the data remains correctly encoded when passed between the database and your Python application.

The ODBC solution

This code ensures that string data is encoded and decoded using UTF-8 when sending and retrieving data to the database. It's especially important when dealing with non-ASCII characters or ensuring compatibility with Unicode data.

def create_connection(connection_string, password):
    connection = None

    try:
        # print(f"Connecting to {connection_string}")
        connection = pyodbc.connect(connection_string   ";PWD="   password)

        # Ensure strings are read correctly
        connection.setdecoding(pyodbc.SQL_CHAR, encoding="utf8")
        connection.setdecoding(pyodbc.SQL_WCHAR, encoding="utf8")
        connection.setencoding(encoding="utf8")

    except pyodbc.Error as e:
        print(f"The error '{e}' occurred")

    return connection

connection.setdecoding(pyodbc.SQL_CHAR, encoding="utf8")

Tells pyodbc how to decode character data from the database when fetching SQL_CHAR types (typically, fixed-length character fields).

connection.setdecoding(pyodbc.SQL_WCHAR, encoding="utf8")

Sets the decoding for SQL_WCHAR, wide-character types (i.e., Unicode strings, such as NVARCHAR or NCHAR in SQL Server).

connection.setencoding(encoding="utf8")

Ensures that any strings or character data sent from Python to the database will be encoded using UTF-8,
meaning Python will translate its internal str type (which is Unicode) into UTF-8 bytes when communicating with the database.

Putting it all together

Install JDBC

Install JAVA - use dmg

https://www.oracle.com/middleeast/java/technologies/downloads/#jdk23-mac

Update shell to set default version

$ /usr/libexec/java_home -V
Matching Java Virtual Machines (2):
    23 (arm64) "Oracle Corporation" - "Java SE 23" /Library/Java/JavaVirtualMachines/jdk-23.jdk/Contents/Home
    1.8.421.09 (arm64) "Oracle Corporation" - "Java" /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home
/Library/Java/JavaVirtualMachines/jdk-23.jdk/Contents/Home
$ echo $SHELL
/opt/homebrew/bin/bash
$ vi ~/.bash_profile

Add JAVA_HOME to your path

export JAVA_HOME=$(/usr/libexec/java_home -v 23)
export PATH=$JAVA_HOME/bin:$PATH

Get the JDBC driver

https://intersystems-community.github.io/iris-driver-distribution/

Put the jar file somewhere... I put it in $HOME

$ ls $HOME/*.jar
/Users/myname/intersystems-jdbc-3.8.4.jar

Sample code

It assumes you have set up ODBC (an example for another day, the dog ate my notes...).

Note: this is a hack of my real code. Note the variable names.

import os

import datetime
from datetime import date, time, datetime, timedelta

import pandas as pd
import pyodbc

import jaydebeapi
import jpype

def jdbc_create_connection(jdbc_url, jdbc_username, jdbc_password):

    # Path to JDBC driver
    jdbc_driver_path = '/Users/yourname/intersystems-jdbc-3.8.4.jar'

    # Ensure JAVA_HOME is set
    os.environ['JAVA_HOME']='/Library/Java/JavaVirtualMachines/jdk-23.jdk/Contents/Home'
    os.environ['CLASSPATH'] = jdbc_driver_path

    # Start the JVM (if not already running)
    if not jpype.isJVMStarted():
        jpype.startJVM(jpype.getDefaultJVMPath(), classpath=[jdbc_driver_path])

    # Connect to the database
    connection = None

    try:
        connection = jaydebeapi.connect("com.intersystems.jdbc.IRISDriver",
                                  jdbc_url,
                                  [jdbc_username, jdbc_password],
                                  jdbc_driver_path)
        print("Connection successful")
    except Exception as e:
        print(f"An error occurred: {e}")

    return connection


def odbc_create_connection(connection_string):
    connection = None

    try:
        # print(f"Connecting to {connection_string}")
        connection = pyodbc.connect(connection_string)

        # Ensure strings are read correctly
        connection.setdecoding(pyodbc.SQL_CHAR, encoding="utf8")
        connection.setdecoding(pyodbc.SQL_WCHAR, encoding="utf8")
        connection.setencoding(encoding="utf8")

    except pyodbc.Error as e:
        print(f"The error '{e}' occurred")

    return connection

# Parameters

odbc_driver = "InterSystems ODBC"
odbc_host = "your_host"
odbc_port = "51773"
odbc_namespace = "your_namespace"
odbc_username = "username"
odbc_password = "password"

jdbc_host = "your_host"
jdbc_port = "51773"
jdbc_namespace = "your_namespace"
jdbc_username = "username"
jdbc_password = "password"

# Create connection and create charts

jdbc_used = True

if jdbc_used:
    print("Using JDBC")
    jdbc_url = f"jdbc:IRIS://{jdbc_host}:{jdbc_port}/{jdbc_namespace}?useUnicode=true&characterEncoding=UTF-8"
    connection = jdbc_create_connection(jdbc_url, jdbc_username, jdbc_password)
else:
    print("Using ODBC")
    connection_string = f"Driver={odbc_driver};Host={odbc_host};Port={odbc_port};Database={odbc_namespace};UID={odbc_username};PWD={odbc_password}"
    connection = odbc_create_connection(connection_string)


if connection is None:
    print("Unable to connect to IRIS")
    exit()

cursor = connection.cursor()

site = "SAMPLE"
table_name = "your.TableNAME"

desired_columns = [
    "RunDate",
    "ActiveUsersCount",
    "EpisodeCountEmergency",
    "EpisodeCountInpatient",
    "EpisodeCountOutpatient",
    "EpisodeCountTotal",
    "AppointmentCount",
    "PrintCountTotal",
    "site",
]

# Construct the column selection part of the query
column_selection = ", ".join(desired_columns)

query_string = f"SELECT {column_selection} FROM {table_name} WHERE Site = '{site}'"

print(query_string)
cursor.execute(query_string)

if jdbc_used:
    # Fetch the results
    results = []
    for row in cursor.fetchall():
        converted_row = [str(item) if isinstance(item, jpype.java.lang.String) else item for item in row]
        results.append(converted_row)

    # Get the column names and ensure they are Python strings (java.lang.String is returned "(p,a,i,n,i,n,t,h,e,a,r,s,e)"
    column_names = [str(col[0]) for col in cursor.description]

    # Create the dataframe
    df = pd.DataFrame.from_records(results, columns=column_names)
    print(df.head().to_string())
else:
    # For very large result sets get results in chunks using cursor.fetchmany(). or fetchall()
    results = cursor.fetchall()
    # Get the column names
    column_names = [column[0] for column in cursor.description]
    # Create the dataframe
    df = pd.DataFrame.from_records(results, columns=column_names)

    print(df.head().to_string())

# # Build charts for a site
# cf.build_7_day_rolling_average_chart(site, cursor, jdbc_used)

cursor.close()
connection.close()

# Shutdown the JVM (if you started it)
# jpype.shutdownJVM()