Project Standards

Objectives

The goal of this document is to provide the developers with clear guidelines to manage their projects.

It is important to emphasize that the usage of specific tools will not be enforced, except for special cases (i.e. docker for containers), but instead the concepts will have to be implemented (i.e. projects must provide unit tests).

To allow the developers to start quickly, generators will be provided when applicable.

Standards

• Service API must be defined using the OpenAPI 3+ specification
  • Zalando provides very good RESTful API Guidelines worth following
• Projects must provide a way to build artifacts using standard tools (i.e. no custom scripts made for a specific machine/environment)
  • a package or binary depending on the language
  • a docker image
• Projects must comply with CI tasks
  • Formatter
  • Linters
    • Code
    • Docstrings
  • Static analyzers
  • Documentation
  • Unit/Feature tests
  • Coverage
    • Reaching 100% coverage is not the end goal
    • Must never regress
    • Think about your teammates who worked on increasing this coverage
    • Use online services like https://coveralls.io or https://codecov.io
    • Use feature tests
      • Try to cover 100% of use cases
        • This is more important the unit test coverage
        • It will end up increasing unit test coverage anywway
• Projects must include a Changelog following the “Keep a Changelog” recommendation (https://keepachangelog.com/en/1.0.0)
• Project must adhere to Semantic Versioning
  • Or Calendar Versioning if it makes more sense.
• Projects must use Issues and Pull Request templates
• Projects must use meaningful labels
• Projects must be documented
• Projects must automate all their maintenance tasks
  • Must ensure reproducibility
  • A Makefile or a package.json at the root of the project is a good example of how to automate maintenance tasks. The commands must be backed by a set of bash scripts (instead of being written inline).

Recommendation per language

Recommendations are provided to help the developers choose the right tool for the right task without spending too much time on the research phase.

All

• Changelog
• Project templates
  • Github
    • Issues
    • Pull Requests
    • Contributing
• EditorConfig to standardize editor configurations.

Python

• Project source layout

.
├── CHANGELOG.md
├── Dockerfile
├── LICENSE
├── README.rst
├── assets
│   └── logos
├── code-of-conduct.md
├── contrib
│   └── scrapd-complete.sh
├── contributors.md
├── docs
│   └── source
├── noxfile.py
├── requirements-dev.txt
├── requirements.txt
├── scrapd
│   ├── __init__.py
│   ├── cli
│   ├── core
│   └── main.py
├── setup.cfg
├── setup.py
├── tasks.py
└── tests
    ├── __init__.py
    ├── core
    ├── data
    ├── features
    ├── mock_data.py
    ├── step_defs
    └── test_common.py

• Packaging
  • Use PBR to create a package
    • Package the application as a wheel (.whl)
  • Alternative: Poetry (does dependency+package management)
• CI tasks
  • Formatter: Black
    • Alternative: YAPF
  • Linters
    • pycodestyle
    • pydocstyle
  • Static analyzers
    • PyFlakes
    • Pylint
    • McCabe’s script
  • Documentation: Sphinx
  • Unit/Feature tests
    • Pytest with the following plugins
      • pytest-asyncio
      • pytest-bdd
      • pytest-cov
      • pytest-mock
      • pytest-rerunfailures
      • pytest-socket
      • pytest-xdist
    • Coverage: coverage.py
• Maintenance tasks automation: Invoke + Nox
  • Alternative: Makefile
  • Alternative: Tox

Remarks

• Flake8 can be used as a wrapper for PyFlakes, pycodestyle, and Ned Batchelder’s McCabe script.

Node.js/React

• CI Tasks
  • Formatter: Prettier
  • Linters: ESLint
  • Unit tests:
    • Framework: Jest
    • Coverage: Jest
• Maintenance tasks automation
  • package.json: https://github.com/scrapd/scrapdviz/blob/master/package.json

Go

• Formatter (built-in): goimports
• Linters: golangci-lint
• Documentation (built-in): godoc
• Unit tests:
  • Framework: go test (built-in): go test -v -cover -coverprofile=coverage.out ./...
  • Coverage: go tool (built-in): go tool cover -html=coverage.out
• Maintenance tasks automation
  • Go Task
  • Alternative: Mage

Ruby

• Formatter: rubocop
• Linters: rubocop
• Static analyzers: rubocop
• Documentation: yard
• Unit test: rspec
• Coverage: simplecov