Plugin system in python

Creating and discovering plugins¶

Often when creating a Python application or library you’ll want the ability to provide customizations or extra features via plugins. Because Python packages can be separately distributed, your application or library may want to automatically discover all of the plugins available.

There are three major approaches to doing automatic plugin discovery:

Using naming convention¶

If all of the plugins for your application follow the same naming convention, you can use pkgutil.iter_modules() to discover all of the top-level modules that match the naming convention. For example, Flask uses the naming convention flask_ . If you wanted to automatically discover all of the Flask plugins installed:

import importlib import pkgutil discovered_plugins =  name: importlib.import_module(name) for finder, name, ispkg in pkgutil.iter_modules() if name.startswith('flask_') > 

If you had both the Flask-SQLAlchemy and Flask-Talisman plugins installed then discovered_plugins would be:

 'flask_sqlalchemy': module: 'flask_sqlalchemy'>, 'flask_talisman': module: 'flask_talisman'>, > 

Using naming convention for plugins also allows you to query the Python Package Index’s simple repository API for all packages that conform to your naming convention.

Using namespace packages¶

Namespace packages can be used to provide a convention for where to place plugins and also provides a way to perform discovery. For example, if you make the sub-package myapp.plugins a namespace package then other distributions can provide modules and packages to that namespace. Once installed, you can use pkgutil.iter_modules() to discover all modules and packages installed under that namespace:

import importlib import pkgutil import myapp.plugins def iter_namespace(ns_pkg): # Specifying the second argument (prefix) to iter_modules makes the # returned name an absolute name instead of a relative one. This allows # import_module to work without having to do additional modification to # the name. return pkgutil.iter_modules(ns_pkg.__path__, ns_pkg.__name__ + ".") discovered_plugins =  name: importlib.import_module(name) for finder, name, ispkg in iter_namespace(myapp.plugins) > 

Specifying myapp.plugins.__path__ to iter_modules() causes it to only look for the modules directly under that namespace. For example, if you have installed distributions that provide the modules myapp.plugins.a and myapp.plugins.b then discovered_plugins in this case would be:

 'a': module: 'myapp.plugins.a'>, 'b': module: 'myapp.plugins.b'>, > 

This sample uses a sub-package as the namespace package ( myapp.plugins ), but it’s also possible to use a top-level package for this purpose (such as myapp_plugins ). How to pick the namespace to use is a matter of preference, but it’s not recommended to make your project’s main top-level package ( myapp in this case) a namespace package for the purpose of plugins, as one bad plugin could cause the entire namespace to break which would in turn make your project unimportable. For the “namespace sub-package” approach to work, the plugin packages must omit the __init__.py for your top-level package directory ( myapp in this case) and include the namespace-package style __init__.py in the namespace sub-package directory ( myapp/plugins ). This also means that plugins will need to explicitly pass a list of packages to setup() ’s packages argument instead of using setuptools.find_packages() .

Namespace packages are a complex feature and there are several different ways to create them. It’s highly recommended to read the Packaging namespace packages documentation and clearly document which approach is preferred for plugins to your project.

Using package metadata¶

Packages can have metadata for plugins described in the Entry points specification . By specifying them, a package announces that it contains a specific kind of plugin. Another package supporting this kind of plugin can use the metadata to discover that plugin.

For example if you have a package named myapp-plugin-a and it includes the following in its pyproject.toml :

[project.entry-points.'myapp.plugins'] a = 'myapp_plugin_a' 

Then you can discover and load all of the registered entry points by using importlib.metadata.entry_points() (or the backport importlib_metadata >= 3.6 for Python 3.6-3.9):

import sys if sys.version_info  (3, 10): from importlib_metadata import entry_points else: from importlib.metadata import entry_points discovered_plugins = entry_points(group='myapp.plugins') 

In this example, discovered_plugins would be a collection of type importlib.metadata.EntryPoint :

( EntryPoint(name='a', value='myapp_plugin_a', group='myapp.plugins'), . ) 

Now the module of your choice can be imported by executing discovered_plugins[‘a’].load() .

The entry_point specification in setup.py is fairly flexible and has a lot of options. It’s recommended to read over the entire section on entry points .

Since this specification is part of the standard library , most packaging tools other than setuptools provide support for defining entry points.

Table of Contents

• An Overview of Packaging for Python
• The Packaging Flow
• Tutorials
• Guides
  • Installing packages using pip and virtual environments
  • Installing stand alone command line tools
  • Installing pip/setuptools/wheel with Linux Package Managers
  • Installing scientific packages
  • Package index mirrors and caches
  • Hosting your own simple repository
  • Packaging and distributing projects
  • Including files in source distributions with MANIFEST.in
  • Single-sourcing the package version
  • Dropping support for older Python versions
  • Packaging binary extensions
  • Packaging namespace packages
  • Creating and discovering plugins
  • Using TestPyPI
  • Making a PyPI-friendly README
  • Publishing package distribution releases using GitHub Actions CI/CD workflows
  • Tool recommendations
  • Analyzing PyPI package downloads

  Источник

  Implementing a Plugin Architecture in a Python Application

  In July, I released my first open source project. It’s an apispec plugin that generates OpenAPI Specification (aka Swagger docs) for Falcon web applications.

  Apispec’s design made it easy to extend core functionality for a specific use case. I extended the apispec.BasePlugin class, overrode a couple of methods, and I was done. Had to dig into apispec internals to figure out how things were wired together, but it was easy to build upon what was already there

  This got me thinking about how to implement a plugin system in one of my own applications. Creating a plugin architecture? It sounded hard. An advanced computer science concept, if you will.

  I mulled over a few different implementations based on software I had previously used. I realized this wasn’t a difficult problem. Like everything else in programming, once we deconstruct the problem into smaller chunks, we can reason about implementation details clearly.

  We assume things are more difficult than they appear. This is especially true for problems we have not seen before. Take a step back. Breathe. Break the problem down into smaller pieces. You got this.

  

  In this Quick Hit, we will walk through the implementation of a simple plugin system.

  Background

  A plugin is a software component that adds a specific feature to an existing computer program. When a program supports plug-ins, it enables customization (Wikipedia)

  There are many benefits to building apps with a plugin framework:

  • 3rd party developers can create and extend upon your app
  • new features are easier to develop
  • your application becomes smaller and easier to understand

  Sample Application Flow

  We have a program that starts, does a few things, and exits.

  

  Plugin Architecture Workflow

  We refactored our Business Logic into a Plugin Framework that can run registered plugins. The plugins will need to meet the specifications defined by our framework in order to run.

  

  Toy Example

  Let’s implement a toy example where we have a plugin framework to print to the console. Our project will have the following structure:

  . ├── internal.py # internal business logic ├── external.py # contains user-created plugins └── main.py # initialize and run application 

  Internal

  This module contains the application class and an internal plugin.

  # internal.py class InternalPrinter: """Internal business logic""" def process(self): print("Internal Hello") class MyApplication: """First attempt at a plugin system""" def __init__(self, *, plugins: list=list()): self.internal_modules = [InternalPrinter()] self._plugins = plugins def run(self): print("Starting program") print("-" * 79) modules_to_execute = self.internal_modules + self._plugins for module in modules_to_execute: module.process() print("-" * 79) print("Program done") print() 

  External

  # external.py class HelloWorldPrinter: def process(self): print("Hello World") class AlohaWorldPrinter: def process(self): print("Aloha World") 

  Main

  In this module, we run instances of our application with the external plugins we want to enable.

  # main.py from internal import MyApplication from external import HelloWorldPrinter, AlohaWorldPrinter if __name__ == "__main__": # Run with one plugin app = MyApplication(plugins=[HelloWorldPrinter()]) app.run() # Run with another plugin app = MyApplication(plugins=[AlohaWorldPrinter()]) app.run() # Run with both plugins app = MyApplication(plugins=[HelloWorldPrinter(), AlohaWorldPrinter()]) app.run() 

  Discussion

  The Application’s plugin framework works for both internal and external plugins. We define internal plugins in the application code. External plugins are initialized and passed into the application at runtime.

  Each plugin inherits from the base Python object and has a process() method. Nothing complex, want to keep this example as simple as possible.

  We can run our plugins by calling the application’s run() method. This method loops over all the plugins and calls each instance’s process() function. As we see from the output above, the plugins are executed in the same order as the list.

  Running Toy Application

  $ python main.py Starting program ------------------------------------------------------------------------------- Internal Hello Hello World ------------------------------------------------------------------------------- Program done Starting program ------------------------------------------------------------------------------- Internal Hello Aloha World ------------------------------------------------------------------------------- Program done Starting program ------------------------------------------------------------------------------- Internal Hello Hello World Aloha World ------------------------------------------------------------------------------- Program done 

  Real World Example

  This pattern can be used in conjunction with the Adapter pattern to simplify application development.

  Let’s say we have a large number of external clients we want to interface with. Each API is different, but the tasks we need to perform for each client are the same.

  One possible implementation of this is to write an adapter around each of the client APIs, resulting in a common interface. Next, we can leverage the plugin framework to solve our business problem, and then we can use plugins to make it work for all of our clients.

  This is a very high level description of the solution. I leave implementation as an exercise to the reader.

  Additional Resources

  Источник

  Читайте также:  Set breakpoint in javascript