dashdz

[dashdz]

Fuck Python, Pandas Again

Hi first post in a while, today I tried to load some data with pandas:

$ python -m please.do.something.reasonable Traceback (most recent call last): File "/home/dashdz/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/dashdz/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/dashdz/repos/some-programming-idk/please/do/something/reasonable.py", line 36, in main() File "/home/dashdz/repos/some-programming-idk/please/do/something/reasonable.py", line 20, in main df = pd.read_csv('data/pretty/normal/data.tsv') File "/home/dashdz/.pyenv/versions/please/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f return _read(filepath_or_buffer, kwds) File "/home/dashdz/.pyenv/versions/please/lib/python3.7/site-packages/pandas/io/parsers.py", line 463, in _read data = parser.read(nrows) File "/home/dashdz/.pyenv/versions/please/lib/python3.7/site-packages/pandas/io/parsers.py", line 1154, in read ret = self._engine.read(nrows) File "/home/dashdz/.pyenv/versions/please/lib/python3.7/site-packages/pandas/io/parsers.py", line 2048, in read data = self._reader.read(nrows) File "pandas/_libs/parsers.pyx", line 879, in pandas._libs.parsers.TextReader.read File "pandas/_libs/parsers.pyx", line 894, in pandas._libs.parsers.TextReader._read_low_memory File "pandas/_libs/parsers.pyx", line 948, in pandas._libs.parsers.TextReader._read_rows File "pandas/_libs/parsers.pyx", line 935, in pandas._libs.parsers.TextReader._tokenize_rows File "pandas/_libs/parsers.pyx", line 2130, in pandas._libs.parsers.raise_parser_error pandas.errors.ParserError: Error tokenizing data. C error: Expected 2 fields in line 11, saw 3

But serious question for everyone:

why is normal behavior for well-oiled big mainstream python libraries to crash and vomit a huge stack trace?

I reject we cannot do better.

Here are some alternative error ideas:

$ python -m please.do.something.reasonable ERROR: Using commas (',') to parse the file, everything I've seen so far has 2 fields, such as line 10 of your file ('data/pretty/normal/data.tsv'): hi,there ... but then line 11 of that file ('data/pretty/normal/data.tsv') had 3 fields: why,hello,you this is impossible for me to recover from because of programming, so I have crashed. You've got to make sure that I know how many fields the file is gonna have upfront.

or maybe

$ python -m please.do.something.reasonable ERROR: Despite the docs saying if the separator is None, we will auto detect your delimiter, which kind of implies that maybe the default should be None, when you don't pass it, we assume commas (',') rather than try to detect it. Your file ends with '.tsv', which is kind of a dead giveaway, but whatever, I thought commas was best because then I can use C and GO REAL FAST and oops I crashed.

I mean the whole first person thing kinda creeps me out (TeX or LaTeX or WhAtEvEr does that and I never realize what it means at first), but like u get the picture.

I just don't think using the wrong delimiter and/or just having the wrong number of fields in a line is so crazy of behavior that the right answer is to vomit a stack trace.

[dashdz]

Fuck Python, related work: xkcd

[https://xkcd.com/1987/]

[dashdz]

Fuck Python, with guest appearance by: pandas

import pandas as pd

What could go wrong.

"... automatically detect the separator by Python’s builtin sniffer tool ..." -- (docs)

Cool sounds good

df = pd.read_csv(path) df # not picture here: a mountain of absolute garbage

It's just fucking tabs. OK fine

df = pd.read_csv(path, delimiter='\t')

OK good to go let's see what we've got

>>> df.columns Index(['Vectors', 'comes_in_a_stick', 'does_DIY', 'does_absorb_water', ... 'made_of_wood_and_metal', 'made_of_wool', 'made_of_yeast'], dtype='object', length=2726)

Cool columns works let's check out the first one

>>> df[:,df.columns[0]] Traceback (most recent call last): File "", line 1, in File "python3.6/site-packages/pandas/core/frame.py", line 2139, in __getitem__ return self._getitem_column(key) File "python3.6/site-packages/pandas/core/frame.py", line 2146, in _getitem_column return self._get_item_cache(key) File "python3.6/site-packages/pandas/core/generic.py", line 1840, in _get_item_cache res = cache.get(item) TypeError: unhashable type: 'slice'

Wait that's kind of the whole reason I installed your damn OK fine how about just one cell?

>>> df[0,df.columns[0]] Traceback (most recent call last): File "python3.6/site-packages/pandas/core/indexes/base.py", line 2525, in get_loc return self._engine.get_loc(key) File "pandas/_libs/index.pyx", line 117, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 139, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1265, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1273, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: (0, 'Vectors') During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in File "python3.6/site-packages/pandas/core/frame.py", line 2139, in __getitem__ return self._getitem_column(key) File "python3.6/site-packages/pandas/core/frame.py", line 2146, in _getitem_column return self._get_item_cache(key) File "python3.6/site-packages/pandas/core/generic.py", line 1842, in _get_item_cache values = self._data.get(item) File "python3.6/site-packages/pandas/core/internals.py", line 3843, in get loc = self.items.get_loc(item) File "python3.6/site-packages/pandas/core/indexes/base.py", line 2527, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas/_libs/index.pyx", line 117, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 139, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1265, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1273, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: (0, 'Vectors')

literally how many lines of errors do you have to print out to tell me you haven't implemented SHIT WHY CAN'T I JUST LOOK AT MY ROWS

>>> df.rows Traceback (most recent call last): File "", line 1, in File "python3.6/site-packages/pandas/core/generic.py", line 3614, in __getattr__ return object.__getattribute__(self, name) AttributeError: 'DataFrame' object has no attribute 'rows' >>>

no of course not, that'd be foolish to expect.

[dashdz]

Fuck Python, with guest appearance by: pip

Let's install something with pip

$ pip install something ... You are using pip version 8.1.2, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

OK all right pip let's upgrade you

$ pip install --upgrade pip Collecting pip Using cached pip-9.0.1-py2.py3-none-any.whl Installing collected packages: pip Found existing installation: pip 8.1.2 Uninstalling pip-8.1.2: Exception: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 326, in run strip_file_prefix=options.strip_file_prefix, File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 736, in install requirement.uninstall(auto_confirm=True) File "/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 742, in uninstall paths_to_remove.remove(auto_confirm) File "/usr/lib/python2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove renames(path, new_path) File "/usr/lib/python2.7/site-packages/pip/utils/__init__.py", line 267, in renames shutil.move(old, new) File "/usr/lib64/python2.7/shutil.py", line 302, in move os.unlink(src) OSError: [Errno 13] Permission denied: '/usr/bin/pip' You are using pip version 8.1.2, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

OK I guess I need to be root oh wait I can't

$ sudo -bash: sudo: command not found $ su Password: su: Authentication failure $ ksu account root: authorization failed

Don't worry there's user mode

$ pip install --user --upgrade pip Collecting pip Using cached pip-9.0.1-py2.py3-none-any.whl Installing collected packages: pip Successfully installed pip-8.1.2 You are using pip version 8.1.2, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

Wait what

$ pip install --upgrade --user pip Collecting pip Using cached pip-9.0.1-py2.py3-none-any.whl Installing collected packages: pip Successfully installed pip-8.1.2 You are using pip version 8.1.2, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

Cool yep well yeah good glad this is working

[dashdz]

Fuck Python, part 2

Oh, so you want to use some other code you wrote in python? Well, FUCK YOU.

try figuring out how python imports shit

change PYTHONCRAP environment variables everywhere and feel guilty about it and that there must be a better way

start splattering __init__.pys everywhere

try to figure out how to "package" your python shit in pip and pypi (and don't you dare fucking get that confused with pypy which is god knows what) but then realize you don't want to publish it or distribute it or anything you just want to import it in another fucking script on your same computer and NO I DON'T WANT TO HOST MY OWN PYPI REPOSITORY YES I DON'T CARE THAT IT'S EASY

try using ln -s to paste shit around

vomit on your laptop and go home

[dashdz]

Fuck Python

Python is so fucked. All I want to use is numpy and matplotlib.

You get yourself fucked with your global system-wide python configuration.

So you use virtualenv to define a virtualenv per project. Fine.

You read this and find out about virtualenvwrapper and autoenv and virutalenv-burrito whose bugs (autoenv: a buggy overriding of cd? Are you joking?) and ludicrously long names (is virtualenv-burrito the best we can do for fucking writing some god damn scripts?) make you steer clear.

Then you discover that your virtualenvs, even per project, are getting fucked because you don't have the right versions of python installed, so you use pyenv.

Oh, but wait, you still have to have a virutalenv per project, but pyenv now has to know about it, so you install pyenv-virtualenv (using brew, of course) and pray that the jerry-rigged, unholy mess that must now be your bash config somehow pulls through.

You cover your eyes at the fact that pyenv-virtualenvwrapper exists.

You try to import matplotlib.dates but discover an issue (about which you ignore the fact that the pyenv developer responded with a middle finger and closed it) and fucking have to recompile python using the invocation some commenter pasted, which somehow works.

You then, finally, get to fucking run matplotlib but then, lo and behold, matplotlib shits the bed because it doesn't work with virtual environments (which the web page matter-of-factly notes isn't matplotlib's fault, but rather the GUI libraries on which it relies, blah blah blah), and I literally just hallucinated the smell of dog shit while writing this.

Your brain searches for simple solutions because you just want to run a god damned python script using extremely popular libraries and why god why so you consider your "system python" but that got fucked a while back because you had to recompile from source because the distribution it didn't fully support UTF-8 which you needed and you couldn't even find the goddamned binary so had to globally install it and it, the system python, now exists in probably three or four places given OSX and Homebrew's python schizophrenia along with your compilation woes.

You consider learning Julia or R or god forbid buying MATLAB or god forbid forbid javascript because well at least npm and nvm have solved all of this bullshit and realize that there's nothing left in this world but the cold dark configuration hell of PYTHONTHIS and PYTHONTHAT (and don't forget THISPYTHON and THATPYTHON too) and you consider whether it might be worth switching fields because surely painters and theoretical computer scientists don't have to deal with this mess.

[dashdz]

無印良品キャンプ場

from 新村則人

Source: http://whoswho.jagda.org/jp/portfolio/512.html