1.. _tut-io:
2
3****************
4Input and Output
5****************
6
7There are several ways to present the output of a program; data can be printed
8in a human-readable form, or written to a file for future use. This chapter will
9discuss some of the possibilities.
10
11
12.. _tut-formatting:
13
14Fancier Output Formatting
15=========================
16
17So far we've encountered two ways of writing values: *expression statements* and
18the :func:`print` function.  (A third way is using the :meth:`write` method
19of file objects; the standard output file can be referenced as ``sys.stdout``.
20See the Library Reference for more information on this.)
21
22Often you'll want more control over the formatting of your output than simply
23printing space-separated values. There are several ways to format output.
24
25* To use :ref:`formatted string literals <tut-f-strings>`, begin a string
26  with ``f`` or ``F`` before the opening quotation mark or triple quotation mark.
27  Inside this string, you can write a Python expression between ``{`` and ``}``
28  characters that can refer to variables or literal values.
29
30  ::
31
32     >>> year = 2016
33     >>> event = 'Referendum'
34     >>> f'Results of the {year} {event}'
35     'Results of the 2016 Referendum'
36
37* The :meth:`str.format` method of strings requires more manual
38  effort.  You'll still use ``{`` and ``}`` to mark where a variable
39  will be substituted and can provide detailed formatting directives,
40  but you'll also need to provide the information to be formatted.
41
42  ::
43
44     >>> yes_votes = 42_572_654
45     >>> no_votes = 43_132_495
46     >>> percentage = yes_votes / (yes_votes + no_votes)
47     >>> '{:-9} YES votes  {:2.2%}'.format(yes_votes, percentage)
48     ' 42572654 YES votes  49.67%'
49
50* Finally, you can do all the string handling yourself by using string slicing and
51  concatenation operations to create any layout you can imagine.  The
52  string type has some methods that perform useful operations for padding
53  strings to a given column width.
54
55When you don't need fancy output but just want a quick display of some
56variables for debugging purposes, you can convert any value to a string with
57the :func:`repr` or :func:`str` functions.
58
59The :func:`str` function is meant to return representations of values which are
60fairly human-readable, while :func:`repr` is meant to generate representations
61which can be read by the interpreter (or will force a :exc:`SyntaxError` if
62there is no equivalent syntax).  For objects which don't have a particular
63representation for human consumption, :func:`str` will return the same value as
64:func:`repr`.  Many values, such as numbers or structures like lists and
65dictionaries, have the same representation using either function.  Strings, in
66particular, have two distinct representations.
67
68Some examples::
69
70   >>> s = 'Hello, world.'
71   >>> str(s)
72   'Hello, world.'
73   >>> repr(s)
74   "'Hello, world.'"
75   >>> str(1/7)
76   '0.14285714285714285'
77   >>> x = 10 * 3.25
78   >>> y = 200 * 200
79   >>> s = 'The value of x is ' + repr(x) + ', and y is ' + repr(y) + '...'
80   >>> print(s)
81   The value of x is 32.5, and y is 40000...
82   >>> # The repr() of a string adds string quotes and backslashes:
83   ... hello = 'hello, world\n'
84   >>> hellos = repr(hello)
85   >>> print(hellos)
86   'hello, world\n'
87   >>> # The argument to repr() may be any Python object:
88   ... repr((x, y, ('spam', 'eggs')))
89   "(32.5, 40000, ('spam', 'eggs'))"
90
91The :mod:`string` module contains a :class:`~string.Template` class that offers
92yet another way to substitute values into strings, using placeholders like
93``$x`` and replacing them with values from a dictionary, but offers much less
94control of the formatting.
95
96
97.. _tut-f-strings:
98
99Formatted String Literals
100-------------------------
101
102:ref:`Formatted string literals <f-strings>` (also called f-strings for
103short) let you include the value of Python expressions inside a string by
104prefixing the string with ``f`` or ``F`` and writing expressions as
105``{expression}``.
106
107An optional format specifier can follow the expression. This allows greater
108control over how the value is formatted. The following example rounds pi to
109three places after the decimal::
110
111   >>> import math
112   >>> print(f'The value of pi is approximately {math.pi:.3f}.')
113   The value of pi is approximately 3.142.
114
115Passing an integer after the ``':'`` will cause that field to be a minimum
116number of characters wide.  This is useful for making columns line up. ::
117
118   >>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 7678}
119   >>> for name, phone in table.items():
120   ...     print(f'{name:10} ==> {phone:10d}')
121   ...
122   Sjoerd     ==>       4127
123   Jack       ==>       4098
124   Dcab       ==>       7678
125
126Other modifiers can be used to convert the value before it is formatted.
127``'!a'`` applies :func:`ascii`, ``'!s'`` applies :func:`str`, and ``'!r'``
128applies :func:`repr`::
129
130   >>> animals = 'eels'
131   >>> print(f'My hovercraft is full of {animals}.')
132   My hovercraft is full of eels.
133   >>> print(f'My hovercraft is full of {animals!r}.')
134   My hovercraft is full of 'eels'.
135
136The ``=`` specifier can be used to expand an expression to the text of the
137expression, an equal sign, then the representation of the evaluated expression:
138
139   >>> bugs = 'roaches'
140   >>> count = 13
141   >>> area = 'living room'
142   >>> print(f'Debugging {bugs=} {count=} {area=}')
143   Debugging bugs='roaches' count=13 area='living room'
144
145See :ref:`self-documenting expressions <bpo-36817-whatsnew>` for more information
146on the ``=`` specifier. For a reference on these format specifications, see
147the reference guide for the :ref:`formatspec`.
148
149.. _tut-string-format:
150
151The String format() Method
152--------------------------
153
154Basic usage of the :meth:`str.format` method looks like this::
155
156   >>> print('We are the {} who say "{}!"'.format('knights', 'Ni'))
157   We are the knights who say "Ni!"
158
159The brackets and characters within them (called format fields) are replaced with
160the objects passed into the :meth:`str.format` method.  A number in the
161brackets can be used to refer to the position of the object passed into the
162:meth:`str.format` method. ::
163
164   >>> print('{0} and {1}'.format('spam', 'eggs'))
165   spam and eggs
166   >>> print('{1} and {0}'.format('spam', 'eggs'))
167   eggs and spam
168
169If keyword arguments are used in the :meth:`str.format` method, their values
170are referred to by using the name of the argument. ::
171
172   >>> print('This {food} is {adjective}.'.format(
173   ...       food='spam', adjective='absolutely horrible'))
174   This spam is absolutely horrible.
175
176Positional and keyword arguments can be arbitrarily combined::
177
178   >>> print('The story of {0}, {1}, and {other}.'.format('Bill', 'Manfred',
179   ...                                                    other='Georg'))
180   The story of Bill, Manfred, and Georg.
181
182If you have a really long format string that you don't want to split up, it
183would be nice if you could reference the variables to be formatted by name
184instead of by position.  This can be done by simply passing the dict and using
185square brackets ``'[]'`` to access the keys. ::
186
187   >>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678}
188   >>> print('Jack: {0[Jack]:d}; Sjoerd: {0[Sjoerd]:d}; '
189   ...       'Dcab: {0[Dcab]:d}'.format(table))
190   Jack: 4098; Sjoerd: 4127; Dcab: 8637678
191
192This could also be done by passing the ``table`` dictionary as keyword arguments with the ``**``
193notation. ::
194
195   >>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678}
196   >>> print('Jack: {Jack:d}; Sjoerd: {Sjoerd:d}; Dcab: {Dcab:d}'.format(**table))
197   Jack: 4098; Sjoerd: 4127; Dcab: 8637678
198
199This is particularly useful in combination with the built-in function
200:func:`vars`, which returns a dictionary containing all local variables.
201
202As an example, the following lines produce a tidily aligned
203set of columns giving integers and their squares and cubes::
204
205   >>> for x in range(1, 11):
206   ...     print('{0:2d} {1:3d} {2:4d}'.format(x, x*x, x*x*x))
207   ...
208    1   1    1
209    2   4    8
210    3   9   27
211    4  16   64
212    5  25  125
213    6  36  216
214    7  49  343
215    8  64  512
216    9  81  729
217   10 100 1000
218
219For a complete overview of string formatting with :meth:`str.format`, see
220:ref:`formatstrings`.
221
222
223Manual String Formatting
224------------------------
225
226Here's the same table of squares and cubes, formatted manually::
227
228   >>> for x in range(1, 11):
229   ...     print(repr(x).rjust(2), repr(x*x).rjust(3), end=' ')
230   ...     # Note use of 'end' on previous line
231   ...     print(repr(x*x*x).rjust(4))
232   ...
233    1   1    1
234    2   4    8
235    3   9   27
236    4  16   64
237    5  25  125
238    6  36  216
239    7  49  343
240    8  64  512
241    9  81  729
242   10 100 1000
243
244(Note that the one space between each column was added by the
245way :func:`print` works: it always adds spaces between its arguments.)
246
247The :meth:`str.rjust` method of string objects right-justifies a string in a
248field of a given width by padding it with spaces on the left. There are
249similar methods :meth:`str.ljust` and :meth:`str.center`. These methods do
250not write anything, they just return a new string. If the input string is too
251long, they don't truncate it, but return it unchanged; this will mess up your
252column lay-out but that's usually better than the alternative, which would be
253lying about a value. (If you really want truncation you can always add a
254slice operation, as in ``x.ljust(n)[:n]``.)
255
256There is another method, :meth:`str.zfill`, which pads a numeric string on the
257left with zeros.  It understands about plus and minus signs::
258
259   >>> '12'.zfill(5)
260   '00012'
261   >>> '-3.14'.zfill(7)
262   '-003.14'
263   >>> '3.14159265359'.zfill(5)
264   '3.14159265359'
265
266
267Old string formatting
268---------------------
269
270The % operator (modulo) can also be used for string formatting. Given ``'string'
271% values``, instances of ``%`` in ``string`` are replaced with zero or more
272elements of ``values``. This operation is commonly known as string
273interpolation. For example::
274
275   >>> import math
276   >>> print('The value of pi is approximately %5.3f.' % math.pi)
277   The value of pi is approximately 3.142.
278
279More information can be found in the :ref:`old-string-formatting` section.
280
281
282.. _tut-files:
283
284Reading and Writing Files
285=========================
286
287.. index::
288   pair: built-in function; open
289   pair: object; file
290
291:func:`open` returns a :term:`file object`, and is most commonly used with
292two positional arguments and one keyword argument:
293``open(filename, mode, encoding=None)``
294
295::
296
297   >>> f = open('workfile', 'w', encoding="utf-8")
298
299.. XXX str(f) is <io.TextIOWrapper object at 0x82e8dc4>
300
301   >>> print(f)
302   <open file 'workfile', mode 'w' at 80a0960>
303
304The first argument is a string containing the filename.  The second argument is
305another string containing a few characters describing the way in which the file
306will be used.  *mode* can be ``'r'`` when the file will only be read, ``'w'``
307for only writing (an existing file with the same name will be erased), and
308``'a'`` opens the file for appending; any data written to the file is
309automatically added to the end.  ``'r+'`` opens the file for both reading and
310writing. The *mode* argument is optional; ``'r'`` will be assumed if it's
311omitted.
312
313Normally, files are opened in :dfn:`text mode`, that means, you read and write
314strings from and to the file, which are encoded in a specific *encoding*.
315If *encoding* is not specified, the default is platform dependent
316(see :func:`open`).
317Because UTF-8 is the modern de-facto standard, ``encoding="utf-8"`` is
318recommended unless you know that you need to use a different encoding.
319Appending a ``'b'`` to the mode opens the file in :dfn:`binary mode`.
320Binary mode data is read and written as :class:`bytes` objects.
321You can not specify *encoding* when opening file in binary mode.
322
323In text mode, the default when reading is to convert platform-specific line
324endings (``\n`` on Unix, ``\r\n`` on Windows) to just ``\n``.  When writing in
325text mode, the default is to convert occurrences of ``\n`` back to
326platform-specific line endings.  This behind-the-scenes modification
327to file data is fine for text files, but will corrupt binary data like that in
328:file:`JPEG` or :file:`EXE` files.  Be very careful to use binary mode when
329reading and writing such files.
330
331It is good practice to use the :keyword:`with` keyword when dealing
332with file objects.  The advantage is that the file is properly closed
333after its suite finishes, even if an exception is raised at some
334point.  Using :keyword:`!with` is also much shorter than writing
335equivalent :keyword:`try`\ -\ :keyword:`finally` blocks::
336
337    >>> with open('workfile', encoding="utf-8") as f:
338    ...     read_data = f.read()
339
340    >>> # We can check that the file has been automatically closed.
341    >>> f.closed
342    True
343
344If you're not using the :keyword:`with` keyword, then you should call
345``f.close()`` to close the file and immediately free up any system
346resources used by it.
347
348.. warning::
349   Calling ``f.write()`` without using the :keyword:`!with` keyword or calling
350   ``f.close()`` **might** result in the arguments
351   of ``f.write()`` not being completely written to the disk, even if the
352   program exits successfully.
353
354..
355   See also https://bugs.python.org/issue17852
356
357After a file object is closed, either by a :keyword:`with` statement
358or by calling ``f.close()``, attempts to use the file object will
359automatically fail. ::
360
361   >>> f.close()
362   >>> f.read()
363   Traceback (most recent call last):
364     File "<stdin>", line 1, in <module>
365   ValueError: I/O operation on closed file.
366
367
368.. _tut-filemethods:
369
370Methods of File Objects
371-----------------------
372
373The rest of the examples in this section will assume that a file object called
374``f`` has already been created.
375
376To read a file's contents, call ``f.read(size)``, which reads some quantity of
377data and returns it as a string (in text mode) or bytes object (in binary mode).
378*size* is an optional numeric argument.  When *size* is omitted or negative, the
379entire contents of the file will be read and returned; it's your problem if the
380file is twice as large as your machine's memory. Otherwise, at most *size*
381characters (in text mode) or *size* bytes (in binary mode) are read and returned.
382If the end of the file has been reached, ``f.read()`` will return an empty
383string (``''``).  ::
384
385   >>> f.read()
386   'This is the entire file.\n'
387   >>> f.read()
388   ''
389
390``f.readline()`` reads a single line from the file; a newline character (``\n``)
391is left at the end of the string, and is only omitted on the last line of the
392file if the file doesn't end in a newline.  This makes the return value
393unambiguous; if ``f.readline()`` returns an empty string, the end of the file
394has been reached, while a blank line is represented by ``'\n'``, a string
395containing only a single newline.  ::
396
397   >>> f.readline()
398   'This is the first line of the file.\n'
399   >>> f.readline()
400   'Second line of the file\n'
401   >>> f.readline()
402   ''
403
404For reading lines from a file, you can loop over the file object. This is memory
405efficient, fast, and leads to simple code::
406
407   >>> for line in f:
408   ...     print(line, end='')
409   ...
410   This is the first line of the file.
411   Second line of the file
412
413If you want to read all the lines of a file in a list you can also use
414``list(f)`` or ``f.readlines()``.
415
416``f.write(string)`` writes the contents of *string* to the file, returning
417the number of characters written. ::
418
419   >>> f.write('This is a test\n')
420   15
421
422Other types of objects need to be converted -- either to a string (in text mode)
423or a bytes object (in binary mode) -- before writing them::
424
425   >>> value = ('the answer', 42)
426   >>> s = str(value)  # convert the tuple to string
427   >>> f.write(s)
428   18
429
430``f.tell()`` returns an integer giving the file object's current position in the file
431represented as number of bytes from the beginning of the file when in binary mode and
432an opaque number when in text mode.
433
434To change the file object's position, use ``f.seek(offset, whence)``.  The position is computed
435from adding *offset* to a reference point; the reference point is selected by
436the *whence* argument.  A *whence* value of 0 measures from the beginning
437of the file, 1 uses the current file position, and 2 uses the end of the file as
438the reference point.  *whence* can be omitted and defaults to 0, using the
439beginning of the file as the reference point. ::
440
441   >>> f = open('workfile', 'rb+')
442   >>> f.write(b'0123456789abcdef')
443   16
444   >>> f.seek(5)      # Go to the 6th byte in the file
445   5
446   >>> f.read(1)
447   b'5'
448   >>> f.seek(-3, 2)  # Go to the 3rd byte before the end
449   13
450   >>> f.read(1)
451   b'd'
452
453In text files (those opened without a ``b`` in the mode string), only seeks
454relative to the beginning of the file are allowed (the exception being seeking
455to the very file end with ``seek(0, 2)``) and the only valid *offset* values are
456those returned from the ``f.tell()``, or zero. Any other *offset* value produces
457undefined behaviour.
458
459File objects have some additional methods, such as :meth:`~file.isatty` and
460:meth:`~file.truncate` which are less frequently used; consult the Library
461Reference for a complete guide to file objects.
462
463
464.. _tut-json:
465
466Saving structured data with :mod:`json`
467---------------------------------------
468
469.. index:: pair: module; json
470
471Strings can easily be written to and read from a file.  Numbers take a bit more
472effort, since the :meth:`read` method only returns strings, which will have to
473be passed to a function like :func:`int`, which takes a string like ``'123'``
474and returns its numeric value 123.  When you want to save more complex data
475types like nested lists and dictionaries, parsing and serializing by hand
476becomes complicated.
477
478Rather than having users constantly writing and debugging code to save
479complicated data types to files, Python allows you to use the popular data
480interchange format called `JSON (JavaScript Object Notation)
481<https://json.org>`_.  The standard module called :mod:`json` can take Python
482data hierarchies, and convert them to string representations; this process is
483called :dfn:`serializing`.  Reconstructing the data from the string representation
484is called :dfn:`deserializing`.  Between serializing and deserializing, the
485string representing the object may have been stored in a file or data, or
486sent over a network connection to some distant machine.
487
488.. note::
489   The JSON format is commonly used by modern applications to allow for data
490   exchange.  Many programmers are already familiar with it, which makes
491   it a good choice for interoperability.
492
493If you have an object ``x``, you can view its JSON string representation with a
494simple line of code::
495
496   >>> import json
497   >>> x = [1, 'simple', 'list']
498   >>> json.dumps(x)
499   '[1, "simple", "list"]'
500
501Another variant of the :func:`~json.dumps` function, called :func:`~json.dump`,
502simply serializes the object to a :term:`text file`.  So if ``f`` is a
503:term:`text file` object opened for writing, we can do this::
504
505   json.dump(x, f)
506
507To decode the object again, if ``f`` is a :term:`binary file` or
508:term:`text file` object which has been opened for reading::
509
510   x = json.load(f)
511
512.. note::
513   JSON files must be encoded in UTF-8. Use ``encoding="utf-8"`` when opening
514   JSON file as a :term:`text file` for both of reading and writing.
515
516This simple serialization technique can handle lists and dictionaries, but
517serializing arbitrary class instances in JSON requires a bit of extra effort.
518The reference for the :mod:`json` module contains an explanation of this.
519
520.. seealso::
521
522   :mod:`pickle` - the pickle module
523
524   Contrary to :ref:`JSON <tut-json>`, *pickle* is a protocol which allows
525   the serialization of arbitrarily complex Python objects.  As such, it is
526   specific to Python and cannot be used to communicate with applications
527   written in other languages.  It is also insecure by default:
528   deserializing pickle data coming from an untrusted source can execute
529   arbitrary code, if the data was crafted by a skilled attacker.
530