doris/fe/fe-extension-loader

fe-extension-loader Developer Integration Guide

This guide is for business module developers. It focuses on:

1. What Loader solves.
2. When to use it.
3. What each core class is responsible for.
4. How to integrate it in business modules.
5. How to diagnose common failures.

1. Module Positioning

fe-extension-loader is the reusable plugin runtime foundation for Doris FE.

It unifies repeated loading logic across modules, including:

1. Scanning pluginRoots.
2. Resolving jars under plugin directories.
3. Building child-first classloaders.
4. Discovering typed factories via ServiceLoader.
5. Aggregating load successes and failures.

2. Applicable Scenarios

2.1 Good Fit For Loader

Use Loader if your module needs:

1. External plugin loading from directories.
2. Shared loading framework across multiple business modules.
3. Standardized failure semantics (scan, resolve, discover, etc.).
4. Less duplicated runtime loading code.

2.2 Not A Good Fit For Loader

Loader may not be a direct fit if:

1. Plugin source is not directory-based and you already have a custom loading pipeline.
2. You only need SPI contracts and no runtime loading.

In those cases, use fe-extension-spi only.

3. Core Class Guide

3.1 DirectoryPluginRuntimeManager<F>

This is the runtime entry class and unified facade.

Primary methods:

1. loadAll(...)
2. get(pluginName)
3. list()

Business modules should depend on this class directly.

3.2 Core Flow Inside DirectoryPluginRuntimeManager

The manager performs:

1. Scan plugin subdirectories under each root.
2. Collect jars from pluginDir/*.jar and pluginDir/lib/*.jar.
3. Create classloader.
4. Discover and validate factory (exactly one per directory).
5. Handle duplicate names and record failures.
6. Return LoadReport.

3.3 PluginLoader

Low-level utility class. It does not scan directories. It only handles:

1. Classloader creation.
2. Typed factory discovery.

Use it when you already own classloader lifecycle externally.

3.4 ChildFirstClassLoader

Classloading behavior:

1. Child-first by default.
2. Parent-first for allowlisted package prefixes.

Purpose:

1. Reduce dependency conflict with FE process classpath.
2. Keep SPI interface class source consistent and avoid type-isolation ClassCastException.

3.5 ClassLoadingPolicy

Used to configure parent-first prefixes:

1. Mandatory prefixes (always included by default).
2. Business prefixes (append per module).

Common business examples:

1. org.apache.doris.authentication.
2. org.apache.doris.authorization.

3.6 PluginHandle<F>

Represents one successfully loaded plugin, including:

1. pluginName
2. pluginDir
3. resolvedJars
4. classLoader
5. factory
6. loadedAt

Business modules typically consume pluginName + factory for registration.

3.7 LoadFailure

Represents one failed plugin directory load, including:

1. pluginDir
2. stage
3. message
4. cause

Failure stages:

1. scan
2. resolve
3. createClassLoader
4. discover
5. instantiate
6. conflict

3.8 LoadReport<F>

Represents the full result of one loadAll call, including:

1. Success list: successes
2. Failure list: failures
3. Statistics: rootsScanned, dirsScanned

3.9 Runtime Handle Storage (Built-In)

DirectoryPluginRuntimeManager stores loaded handles in an internal concurrent map.

No separate PluginRuntimeRegistry abstraction is exposed in current implementation.

4. Standard Integration Steps

4.1 Prepare Typed Factory

Your business factory interface should extend PluginFactory.

4.2 Initialize Runtime Manager

DirectoryPluginRuntimeManager<MyPluginFactory> runtime =
        new DirectoryPluginRuntimeManager<>();

4.3 Configure Classloading Policy

ClassLoadingPolicy policy = new ClassLoadingPolicy(
        Collections.singletonList("org.apache.doris.mybiz."));

4.4 Call loadAll

LoadReport<MyPluginFactory> report = runtime.loadAll(
        pluginRoots,
        Thread.currentThread().getContextClassLoader(),
        MyPluginFactory.class,
        policy);

4.5 Process Load Result

for (LoadFailure failure : report.getFailures()) {
    LOG.warn("plugin load failure: dir={}, stage={}, message={}",
            failure.getPluginDir(), failure.getStage(), failure.getMessage(), failure.getCause());
}

for (PluginHandle<MyPluginFactory> handle : report.getSuccesses()) {
    factoryMap.putIfAbsent(handle.getPluginName(), handle.getFactory());
}

5. Directory Convention

Recommended layout:

<pluginRoot>/
  <pluginA>/
    pluginA.jar
    lib/
      dep1.jar
      dep2.jar
  <pluginB>/
    pluginB.jar

Rules:

1. Only direct subdirectories under pluginRoot are scanned.
2. Each plugin directory must contain at least one jar.
3. Each plugin directory must discover exactly one factory.

6. Conflict And Failure Handling

6.1 Duplicate Plugin Name

Current default strategy:

1. Keep the first successfully loaded plugin.
2. Record later duplicates as conflict.
3. Continue loading other directories.

Business recommendations:

1. Treat conflict as warning/alert.
2. Avoid duplicate plugin names in production plugin roots.

6.2 Should All Failures Throw

Loader returns LoadReport and does not force exception.

Business modules choose policy by semantics:

1. Tolerant mode: log failures and continue startup.
2. Strict mode: fail-fast when no successful plugin exists.

6.3 Detailed LoadReport Handling Strategy

LoadReport should be startup decision input, not just logs.

Recommended goals:

1. All successful plugins are registered.
2. Every failure is traceable by stage and directory.
3. Startup behavior is deterministic (strict or tolerant).
4. Conflict/failure paths do not leak resources.

Recommended processing order:

1. Log summary metrics: rootsScanned, dirsScanned, successes.size, failures.size.
2. Log each failure with pluginDir + stage + message + cause.
3. Register successful plugins to business map (pluginName -> factory).
4. Apply startup decision logic (strict/tolerant).

Suggested stage severity grouping:

1. Environment/config issues: scan, resolve.
2. SPI/implementation issues: discover, instantiate.
3. Runtime construction issues: createClassLoader.
4. Naming conflict: conflict (usually warning, not immediate stop).

Recommended decision rules:

1. dirsScanned == 0: often means empty roots or no external setup.
2. dirsScanned > 0 && successes.isEmpty() in strict mode: fail-fast.
3. dirsScanned > 0 && successes.isEmpty() in tolerant mode: warn and continue only if business allows no external plugin.
4. If requiredPluginNames exists, enforce presence even when partial loads succeeded.

6.4 Copy-And-Adapt Template

public static <F extends PluginFactory> void processLoadReport(
        LoadReport<F> report,
        Map<String, F> factoryMap,
        boolean strictMode,
        Set<String> requiredPluginNames) {
    Objects.requireNonNull(report, "report");
    Objects.requireNonNull(factoryMap, "factoryMap");
    Objects.requireNonNull(requiredPluginNames, "requiredPluginNames");

    // Step 1: summary metrics
    LOG.info("plugin load summary: rootsScanned={}, dirsScanned={}, successCount={}, failureCount={}",
            report.getRootsScanned(),
            report.getDirsScanned(),
            report.getSuccesses().size(),
            report.getFailures().size());

    // Step 2: failure details
    LoadFailure firstNonConflictFailure = null;
    for (LoadFailure failure : report.getFailures()) {
        LOG.warn("plugin load failure: dir={}, stage={}, message={}",
                failure.getPluginDir(), failure.getStage(), failure.getMessage(), failure.getCause());
        if (!LoadFailure.STAGE_CONFLICT.equals(failure.getStage()) && firstNonConflictFailure == null) {
            firstNonConflictFailure = failure;
        }
    }

    // Step 3: register successful plugins
    int registered = 0;
    for (PluginHandle<F> handle : report.getSuccesses()) {
        F existing = factoryMap.putIfAbsent(handle.getPluginName(), handle.getFactory());
        if (existing != null) {
            // If business map already contains the name, close discarded external classloader.
            closeClassLoaderQuietly(handle.getClassLoader());
            LOG.warn("skip duplicated plugin name in business map: {}", handle.getPluginName());
            continue;
        }
        registered++;
    }

    // Step 4: startup decision (strict/tolerant)
    if (strictMode && report.getDirsScanned() > 0 && registered == 0 && firstNonConflictFailure != null) {
        throw new IllegalStateException(
                "No plugin loaded in strict mode: stage=" + firstNonConflictFailure.getStage()
                        + ", dir=" + firstNonConflictFailure.getPluginDir()
                        + ", message=" + firstNonConflictFailure.getMessage(),
                firstNonConflictFailure.getCause());
    }

    // Step 5: required plugin checks
    for (String required : requiredPluginNames) {
        if (!factoryMap.containsKey(required)) {
            throw new IllegalStateException("Required plugin is missing: " + required);
        }
    }
}

The closeClassLoaderQuietly implementation pattern can be referenced from:

../fe-authentication/fe-authentication-handler/src/main/java/org/apache/doris/authentication/handler/AuthenticationPluginManager.java

6.5 Authentication Module Mapping

Current authentication module handling is:

1. Iterate report.getFailures() and log warnings.
2. Iterate report.getSuccesses() and register factories (close duplicated external classloader if needed).
3. Throw AuthenticationException when directories were scanned but no external plugin was loaded.

Reference implementation:

../fe-authentication/fe-authentication-handler/src/main/java/org/apache/doris/authentication/handler/AuthenticationPluginManager.java

7. Current Runtime Scope (V1)

Supported:

1. loadAll
2. get
3. list

Not supported:

1. reload
2. unload

Do not depend on runtime hot-reload semantics in V1.

8. Frequently Asked Questions

8.1 Factory Not Found

Check:

1. Plugin jar includes META-INF/services/<factoryType>.
2. Service file class name is correct.
3. Provider class is included in final jar.

8.2 Multiple Factories Found

Check:

1. Multiple jars may declare the same factory type.
2. Parent classpath may pollute service resources.

Note:

DirectoryPluginRuntimeManager includes parent service-resource filtering and prefers plugin-directory-local discovery.

8.3 Loaded Plugin Not Released

Check:

1. Business layer may keep stale handle references.
2. Conflict/failure paths may skip classloader close.
3. Plugin instance close() may not release resources.

9. Integration With Authentication Module

Authentication integration sample:

../fe-authentication/fe-authentication-handler/src/main/java/org/apache/doris/authentication/handler/AuthenticationPluginManager.java

Key integration points:

1. Use DirectoryPluginRuntimeManager<AuthenticationPluginFactory>.
2. Append authentication parent-first prefix.
3. Register successful factories into authentication factory map.
4. Close classloader for discarded conflicting handles.

Related Docs

1. Chinese developer guide: README_CN.md
2. Unified runtime design (CN): ../fe-authentication/EXTENSION_LOADER_UNIFIED_DESIGN_CN.md
3. SPI contracts: ../fe-extension-spi/README.md