What’s Driving Python’s Massive Popularity?

(Funtap/Shutterstock)

Earlier this month, Python moved into the number one slot in the TIOBE Index, marking the first time in 20 years that a language named C or Java wasn’t at the top of the list. It’s a nice feather in Python’s cap, and marks the peak of the language’s reach. But what’s behind Python’s awesome success?

Python started out modestly enough, as a hobby to keep Guido van Rossum busy through the 1989 holidays. Little did he know he was unleashing a programming language that would have a lasting impact on a segment of the IT world for more than 30 years.

Van Rossum’s created Python to be an interpreted general purpose programming language. His original design goals for Python were that it would be an easy and intuitive language to learn, as understandable as plain English, suitable for everyday tasks, and open source.

The legacy of those designs goals is plainly evident when one talks with members of the Python community, which is an increasingly wide group of individuals who are using Python for a wide variety of things. While the programmers ostensibly are writing Python for work, one gets the feeling that they get real enjoyment out of it.

“I chose Python because it was a language that was fun to code in,” says Calvin Hendryx-Parker, the CTO and co-founder of Six Feet Up, a Fishers, Indiana-based consultancy that does a lot of custom Python coding.

“I enjoy it because it’s not doing just grunt work,” he continues. “It’s all very innovative and creative. It’s more of a craftsman-style skill. Engineers build the same thing day in day out, but craftsman have to solve problems in creative ways because the environment around them is constantly moving.”

Python moved into the number one spot on the TIOBE Index of langauges (Source: TIOBE Index)

Six Feet Up was founded to build custom Web applications, but customers throughout the Midwest are increasingly tapping the firm for help with bringing big data projects to production. While Python isn’t the only language involved in these projects, they typically do involve a healthy dose of everybody’s favorite scripting language.

“What’s nice is a lot of the big data tooling is starting to be built in Python,” Hendryx-Parker says. “It’s this kind of lingua franca for all things data.”

The Massive Python Ecosystem

The Python data ecosystem is wide and growing. From the Python wrappers for Keras and TensorFlow to machine learning packages like scikit-learn and NumPy, from Python interfaces for Spark and Dask clusters to experimenting with code in a Jupyter notebooks or Pandas, data scientists and engineers can do practically anything they need without leaving Python’s comfortable syntax.

The flexibility of Python is front and center for Prefect, a Washington, D.C.-based company that develops a data pipeline tool based on Python. According to Prefect CTO Chris White, Python is the perfect tool for the job.

Prefect CTO Chris White appreciates the flexibility of Python

“At a high level, our design sensibility is sensible defaults but deeply configurable, so you can kind of get something running really quickly if you really want to, but when you really want to get in the weeds, you can also do that,” White says. “Python really helps us achieve that goal, because at the end of the day, Python is a very flexible language. You can write a script. You don’t need to know what compiling means or anything. You just write a script.”

It all points back to van Rossum’s design ethos for Python, which was simplicity over complexity. The barrier to entry for Python is incredibly low, thanks to its imminent readability. That allows Prefect to use Python with its junior coders and more experienced programmers alike.

“It’s one of the few languages, at least I know of, that can be used as a raw scripting language. You just run it. You don’t have to know anything else,” White says. “So it’s very accessible. and then of course you can package things up. You can write C extensions and things like that.”

Flexibility and Performance

Speaking of C, performance is not one of Python’s strengths. That is one of the tradeoffs for imminent flexibility and approachability. Thanks to the C extensions, there is a path to achieve higher performance. But in many cases, that performance is not needed.

“There are other programming languages of course that are going to be a lot faster than Python in those most extreme edge cases, for real-time processing or something,” White says. “But in 90% of the data that people actually deal with…you don’t need that speed.”

Another historic negative in Python’s repertoire is type safety. Python is a strongly typed language, which has led to criticism by some. However, people have started to realize it’s not as big of a negative as they may have thought, White says.

“A lot of the things that people initially complained about like speed and type safety, people are starting to relax on a little bit. Maybe they aren’t such a big deal for some of the applications that people are dealing with,” he says. “That’s the big thing – you can’t do you a lot of type checking with data. Most data is so unstructured. It’s really hard to even conceive of how you would structure that. But Python makes it really easy to at least start iterating on and figuring out, whereas other languages would kind of force you into a scheme really early.”

Python is favored by data scientists and data engineers (Trismegist san/Shutterstock)

Speed of development is a big plus in Python’s corner. Because the language is so approachable, developers can quickly put something together without turning it into a major project. The casualness of Python coding perhaps may be at its peak in Juypter, where you can execute Python on the spot in the browser, even if things are out of order.

A Force Multiplier

When put into the right hands, Python tools can also function like a force multiplier, delivering sophisticated data engineering and data science capabilities that are beyond what would be possible if we all had to be experts in C or Java. White relays this story of a Prefect customer who used the Prefect tool (a Python creation) to orchestrate some fairly complex data workflows.

“The customer said, ‘I’m not really a software engineer by training ,but I just wrote what I believe to be a highly robust, fault-tolerant batch processing workflow and I deployed it to Kubernetes,’” White says. “He said ‘I have no business knowing what any of those words mean, but Prefect gave me the tools to get there.’”

While Python lowers the barrier to entry, thanks to the flexibility of Python and the enthusiastic user community, one can also write perfectly manicured, top-shelf code that will be an absolutely pleasure for your friendly DevOps engineer to handle.

“Python does let you hammer something out really quickly,” Hendryx-Parker says. “But it also lends itself to easily converting them into a well-shaped software project with all the nice bells and whistles when it comes to unit testing and be able to deploy with CI/CD–all the kind of modern trimmings that come around it.”

The Open Source Community

Another plus in Python’s favor is the open source community that’s been built up around it, according to Hendryx-Parker, who has been participating in Python conferences for nearly 20 years, and who is the founder of the local Python user group in Indianapolis, IndyPy

Calvin Hendryx-Parker is the CTO of Six Feet Up

“It’s a group of amazing people who want to make sure that it’s welcoming and open to other amazing people to come and be a part of this community, which is not always easy in a tech crowd,” he says.

There are thousands of add-ons, additions, and integrations for Python, most of which are open source, but some of which are poorly maintained. That can make it tricky sometimes to avoid Python package that have succumbed to old age or bit rot, Hendryx-Parker says.

On the other hand, when one runs into a useful but neglected library, it also provides an opportunity for Python users to step up and give back to the Python community by volunteering to help maintain the code, he says.

“We are encouraging our customers who are in that situation to have us spend time in their budget improving these open source projects,” Hendryx-Parker says. “I think it’s good all around. It’s a win-win-win. If they can get that budget and they know they need to have that ability in their supply chain, it’s a no brainer.”

Compared to the well-documented ecosystem that has been built up around R, the Python ecosystem is a bit less ordered, which has its pros and cons, White says.

“R has really gotten a lot of traction because it feels kind of like a standard. It has the same universe of packages that all that work really nicely together, and so I think that drives people to R,” he says. “Python, it’s kind of like the Wild West. You can kind of do anything. There’s not a set of packages that are as tightly standardized as the Tidyverse is in R.”

On the flipside, the Python community is so big and so open that users have lots of sources for help. “There’s a lot of good documentation, so you can always get the answers you need very quickly,” White says. “But we do have to go out and look. It’s not quite as self-contained.”

The future for Python is bright. It’s at or near the top of everybody’s list of skills for both data science and data engineering. With its current momentum, Python seems destined to keep growing until something can manage to dislodge it from the top of the heap.

Related Items:

Three Reasons Python Is The AI Lingua Franca

Python Attracts Rookies as Well as Pros

Is Python Strangling R to Death?