How to Efficiently Append One String to Another in Python

In Python, concatenating strings with the ' ' operator is a common task. While the following code is straightforward:

var1 = "foo"
var2 = "bar"
var3 = var1   var2

It raises questions about efficiency, especially for large strings or repeated concatenations.

In-Place String Extension

Fortunately, CPython has implemented an optimization to enhance the efficiency of string concatenation. When only a single reference to a string exists and another string is appended to it, CPython attempts to extend the original string in place. This optimization makes the operation amortized O(n).

For instance, the following code used to be O(n^2):

s = ""
for i in range(n):
    s  = str(i)

However, with the optimization, it now runs in O(n).

Python Implementation Details

Here's an excerpt from the Python C source code that illustrates the optimization:

int
_PyBytes_Resize(PyObject **pv, Py_ssize_t newsize)
{
    /* ... */
    *pv = (PyObject *)
        PyObject_REALLOC((char *)v, PyBytesObject_SIZE   newsize);
    if (*pv == NULL) {
        PyObject_Del(v);
        PyErr_NoMemory();
        return -1;
    }
    _Py_NewReference(*pv);
    sv = (PyBytesObject *) *pv;
    Py_SIZE(sv) = newsize;
    sv->ob_sval[newsize] = '\0';
    sv->ob_shash = -1;          /* invalidate cached hash value */
    return 0;
}

This function allows for the resizing of a string object, but only if there is only one reference to it. The size of the string is changed while preserving the original memory location.

Caution

It's crucial to note that this optimization is not part of the Python specification. It is implemented only in the CPython interpreter. Other Python implementations, such as PyPy or Jython, may exhibit different performance characteristics.

Empirical Testing

Empirically, the optimization is evident in the performance of the following code:

import timeit

s = ""
for i in range(10):
    s  = 'a'

# Time the concatenation of 10 'a' characters
t1 = timeit.timeit(stmt="""s = ""
for i in range(10):
    s  = 'a'""", globals=globals(), number=1000000)

# Time the concatenation of 100 'a' characters
t2 = timeit.timeit(stmt="""s = ""
for i in range(100):
    s  = 'a'""", globals=globals(), number=100000)

# Time the concatenation of 1000 'a' characters
t3 = timeit.timeit(stmt="""s = ""
for i in range(1000):
    s  = 'a'""", globals=globals(), number=10000)

print("10 'a':", t1)
print("100 'a':", t2)
print("1000 'a':", t3)

The results show a significant increase in execution time as the number of concatenations grows, indicating that the optimization is not applicable for larger strings.

Conclusion

While Python's in-place string extension optimization dramatically improves the efficiency of string concatenation in certain scenarios, it's essential to understand the limitations of this implementation. For large strings or when memory management considerations are paramount, alternative methods of string manipulation may be necessary to achieve optimal performance.