Mypy in Python

Preface

Python does not have the ability to explicitly specify variable types. All variable types are inferred based on assignment. This may lead to runtime error such as "str" + 1 (this only produces unintended/unexpected result in JavaScript though), it also prevents editor to offer valuable suggestions.

Mypy is a static type checker. In one of my company’s project, I noticed mypy.ini sitting in the project directory, but not being utilised. After some research on mypy, I decided to add its support together with pre-commit. This blog post discusses mypy usage briefly, but focus more on the configurations of files to be checked or excluded in the context of mypy and pre-commit hook configuration.

Usage

Setup Demo

Demo project structure.

.
├── project
│   ├── pyproject.toml
│   ├── mypy.ini
│   ├── main.py
│   └── module
│       ├── __init__.py
│       ├── coingecko_api.py
│       └── github_api.py
│

Dependency required for the demo.

pyproject.toml

# partial pyproject.toml is shown

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.27.1"

[tool.poetry.dev-dependencies]
mypy = "^0.931"
pre-commit = "^2.17.0"

# ...

Files

./mypy.ini

[mypy]
python_version = 3.9

./main.py

def dummy_func(v: str) -> str:
    return v


if __name__ == "__main__":
    dummy_func("1")

./module/__init__.py

from .coingecko_api import CoinGeckoAPI
from .github_api import get_github_user

__all__ = ["CoinGeckoAPI", "get_github_user"]

./module/coingecko_api.py

import requests


class CoinGeckoAPI:
    def __init__(self, dummy_int: int) -> None:
        self.dummy_str: str = dummy_int

    @property
    def dummy_str(self) -> str:
        return self._dummy_str

    @dummy_str.setter
    def dummy_str(self, dummy_int: int) -> None:
        self._dummy_str = "000" + str(dummy_int)

    def __eq__(self, __o: "CoinGeckoAPI") -> bool:
        return self.dummy_str == __o.dummy_str

    def get_bitcoin_price(self):
        r = requests.get(
            "https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd"
        )

        return r.json()

./module/github_api.py

import requests


def get_github_user():
    r = requests.get("https://api.github.com/user", auth=("user", "pass"))
    return r.status_code

Run type checking

# At project root
mypy .

After running above command, mypy . should complain these 3 types of errors:

1. Library stubs not installed for “requests”
2. Incompatible types in assignment (expression has type “int”, variable has type “str”)
3. Argument 1 of “eq“ is incompatible with supertype “object”; supertype defines the argument type as “object”

Missing library stubs

For this error, mypy already shows Hint: "python3 -m pip install types-requests" to install the missing type dependency, in this case types-requests.

Hints offered are quite accurate in my experience, except the case for grpc for my company’s project, which requires grpc-stubs.

Incompatible type

For this error, mymy is complaining for this line in ./module/coingecko_api.py:

self.dummy_str: str = dummy_int

mypy thinks I try to assign int type to str, despite I used type conversion inside setter annotated by @dummy_str.setter. This is unfortunately not solved at the time of writting, you may watch this issue.

To make mypy happy, we have no choice but to ignore it by:

- self.dummy_str: str = dummy_int
+ self.dummy_str: str = dummy_int # type:ignore

“__eq__()” input type

For this error, mypy again provides a very useful hint message:

def __eq__(self, other: object) -> bool:
    if not isinstance(other, CoinGeckoAPI):
        return NotImplemented
    return <logic to compare two instances>

Configuration

After making relevant changes above, running mypy . should succeed. Now I will discuss about the configuration files.

In order to discuss configurations for files to be checked or ignored, we need to add pre-commit configuration to .pre-commit-config.yaml:

.pre-commit-config.yaml

repos:
- repo: https://github.com/pre-commit/mirrors-mypy
  rev: v0.931 # Use the sha / tag you want to point at
  hooks:
  - id: mypy

Missing stubs again?

After that, running pre-commit run --all-files should report one error about Missing library stubs discussed earlier, despite mypy . reports no error! (Note: You need to add files to version control for pre-commit to work, a simple git add . will work, there is no need to commit the changes though.)

This is because type dependencies need to be added explicitly for mypy hook in .pre-commit-config.yaml:

.pre-commit-config.yaml

repos:
  rev: v0.931 # Use the sha / tag you want to point at
  hooks:
  - id: mypy
+   additional_dependencies:
+   - types-requests==2.27.9 # Note the different version syntax from poetry configuration

After making the changes above, pre-commit run --all-files should report no errors.

Before moving on

For the following sections, please make some changes to break mypy type checking, for me, I did the following evil code:

./main.py

if __name__ == "__main__":
-    dummy_func("1")
+    dummy_func(1)

and

./module/coingecko_api.py

- def __eq__(self, __o: object) -> bool:
+ def __eq__(self, __o: "CoinGeckoAPI") -> bool:
-    if not isinstance(__o, CoinGeckoAPI):
-        return NotImplemented
    return self.dummy_str == __o.dummy_str

Running mypy . and pre-commit run --all-files should both report the same 2 errors now.

Files to check

For mypy to check only module directory:

mypy.ini

[mypy]
python_version = 3.9
+ files = module/

Running mypy now (instead of mypy . earlier) should only report one error from module directory. However, running pre-commit run --all-files still reports two errors. That is because mypy hook in pre-commit does not read files configuration from mypy.ini, we need to make the following changes in .pre-commit-config.yaml:

.pre-commit-config.yaml

repos:
\- repo: https://github.com/pre-commit/mirrors-mypy
  rev: v0.931
  hooks:
  - id: mypy
+   files: ^module/
    additional_dependencies:
    - types-requests==2.27.9

Note the different syntax for files configuration in mypy.ini and mypy hook in .pre-commit-config.yaml. Refer to mypy configuration file for more information about files configuration in mypy.ini; as for .pre-commit-config.yaml, it is just regex, which is much simpler.

(Remember to revert the changes for files configuration made in this section before moving on to the next section.)

Files to exclude

For mypy to exclude module directory:

mypy.ini

[mypy]
python_version = 3.9
+ exclude = module/

To add more directories or files to exclude, the exclude configuration looks like:

mypy.ini

[mypy]
python_version = 3.9
+exclude = (?x)(
+    module/
+    | main.py$ # the `)` could also be put immediately after `$` like `$)`
+ ) # else note this is not a typo, at least one space is needed before `)`, else parsing error occurs

Since everything is excluded, now running mypy . should check no files!

As usual, running pre-commit run --all-files now checks all 4 Python files and report 2 errors, because mypy hook in pre-commit run --all-files does not read exclude configuration in mypy.ini. Add exclude configuration to .pre-commit-config.yaml:

.pre-commit-config.yaml

repos:
\- repo: https://github.com/pre-commit/mirrors-mypy
  rev: v0.931
  hooks:
  - id: mypy
+   exclude: module
    additional_dependencies:
    - types-requests==2.27.9

Running pre-commit run --all-files should only report one error from main.py now. Again, the exclude configuration for mypy hook in pre-commit is using regex ^_^

Other configurations?

From my trial and error, mypy hook respects type check configurations like disallow_any_unimported and per-module options under syntax like[mypy-mymodule].

Conclusion

As one of the comment from this issue:

i’ve got the same problem. it’s annoying to have to have two sources of truth for which files to include (mypy.ini and pre-commit config)

I also found it is annoying to have configurations for the same plugin at different places, but sadly this is the case for mypy.

For me, I would put files, exclude and additional_dependencies configurations in .pre-commit-config.yaml, leave the rest in mypy.ini and always run pre-commit run --all-files to check instead of mypy command. Add pre-commit to git hook scripts locally so that it get executed automatically on git commit and put it as part of the CI process if possible.

Referrence

Why does defining the argument types for __eq__ throw a MyPy type error?

What to do about setters of a different type than their property?

“files” configuration in mypy.ini is ignored for mirrors-mypy

mypy documentation