Refer to:
https://developer.android.com/reference/androidx/test/uiautomator/UiDevice

storePath   File: where the PNG should be written to
scale   float: scale the screenshot down if needed; 1.0f for original size
quality int: quality of the PNG compression; range: 0-100

boolean takeScreenshot (File storePath, 
                float scale, 
                int quality)

But it's using the following code:

Bitmap screenshot = this.getUiAutomation().takeScreenshot();
...
bos = new BufferedOutputStream(new FileOutputStream(storePath));
...
screenshot.compress(CompressFormat.PNG, quality, bos);

While in Bitmap.java:

Write a compressed version of the bitmap to the specified outputstream. If this returns true, the bitmap can be reconstructed by passing a corresponding inputstream to BitmapFactory.decodeStream(). Note: not all Formats support all bitmap configs directly, so it is possible that the returned bitmap from BitmapFactory could be in a different bitdepth, and/or may have lost per-pixel alpha (e.g. JPEG only supports opaque pixels).
Params:
format – The format of the compressed image
quality – Hint to the compressor, 0-100. 0 meaning compress for small size, 100 meaning compress for max quality. Some formats, like PNG which is lossless, will ignore the quality setting
stream – The outputstream to write the compressed data.
Returns:
true if successfully compressed to the specified stream.

public boolean compress(CompressFormat format, int quality, OutputStream stream)

We take screenshots with the format CompressFormat.PNG by default, which then cannot be directly compressed.

So we can make our own method to take screenshots(We can also take the format as an optional param):

    boolean takeScreenshot(String storePath, int quality) {
        UiAutomation automation = InstrumentationRegistry.getInstrumentation().getUiAutomation();
        Bitmap screenshot = automation.takeScreenshot();
        if (screenshot != null) {
            try (BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream("/sdcard/Download/" + storePath + ".jpeg"))) {
                screenshot.compress(Bitmap.CompressFormat.JPEG, quality, bos);
                bos.flush();
                return true;
            } catch (IOException ex) {
                Log.e(TAG, "failed to save screen shot to file", ex);
            } finally {
                screenshot.recycle();
            }

        }
        return false;
    }