PyTorch 2.0 release accelerates open source machine learning

Among the most used machine learning (ML) technologies today is the open source PyTorch framework.

PyTorch Facebook (now known as Meta) launched the 1.0 release in 2016, debuting in 2018. Meta PyTorch project in September 2022 the new PyTorch Foundation, managed by the Linux Foundation. Today, the PyTorch developers took the next big step forward for PyTorch by announcing the first experimental release of PyTorch 2.0. The new release promises to help accelerate ML training and development while maintaining backward compatibility with existing PyTorch application code.

“We’ve added an extra feature called ‘torch.compile’ that users need to include in their codebases,” Soumith Chintala, PyTorch lead maintainer. said VentureBeat. “We’re calling it 2.0 because we think users will find it a significant new addition to the experience.”

A new compiler in PyTorch that makes all the difference for ML

There was discussions about when the PyTorch project called the new release 2.0 in the past.

For example, there was a brief discussion about labeling PyTorch 1.10 as a 2.0 release in 2021. Chintala said that PyTorch 1.10 does not have enough fundamental changes to warrant a major update from 1.9 to 2.0.

The latest general release of PyTorch is version 1.13, released at the end of October. In this release, the main feature came suddenly IBM code contribution enables the machine learning framework to work more effectively with a commodity ethernet-based network for large-scale workloads.

Chintala emphasized that now is the right time for PyTorch 2.0, as the project introduces an additional new paradigm in the PyTorch user experience called torch.compile, which brings users robust speedups not possible in the standard enthusiast mode of PyTorch 1.0. .

He explained that the roughly 160 open-source models that the PyTorch project 2.0 had validated had a 43% speedup, and they worked reliably with a one-line addition to the codebase.

“We expect that with PyTorch 2, people will change the way they use PyTorch every day,” said Chintala.

With PyTorch 2.0, he said, developers will start their experiments with enthusiastic mode and after training their models for a long time, activate compiled mode for extra performance.

“Data scientists will be able to do with PyTorch 2.x what they did with 1.x, but they can do them faster and at greater scale,” Chintala said. “If your model was training for more than 5 days and now trains for 2.5 days with 2.x’s designed mode, then with that extra time you can iterate more ideas or build a larger model that trains for the same 5 days.”

More Python is coming to PyTorch 2.x

PyTorch derives the first part of its name (Py) from the open source Python programming language widely used in data science.

Modern PyTorch releases, however, are not written entirely in Python, as parts of the framework are now written in the C++ programming language.

“Over the years, we have moved many places torch.nn Switch from Python to C++ to squeeze that last mile performance,” Chintala said.

Chintala said that within the next 2.x series (but not in 2.0), the PyTorch project expects to port code related to torch.nn back to Python. He noted that C++ is usually faster than Python, but the new compiler (torch.compile) is faster than running the equivalent code in C++.

“Moving these parts back into Python improves hackability and lowers the barrier for code contributions,” Chintala said.

Work on Python 2.0 will continue over the next few months, and general availability is not expected until March 2023. Along with the development effort, PyTorch has also transitioned from being managed and managed by Meta to its own independent effort.

“It’s early days for the PyTorch Foundation, and you’ll be hearing more on the horizon for much longer,” Chintala said. “The foundation is in the process of implementing various deliverables and defining goals.”