AI & Vectors

Generate image captions using Hugging Face

Use the Hugging Face Inference API to make calls to 100,000+ Machine Learning models from Supabase Edge Functions.

We can combine Hugging Face with Supabase Storage and Database Webhooks to automatically caption for any image we upload to a storage bucket.

About Hugging Face

Hugging Face is the collaboration platform for the machine learning community.

Huggingface.js provides a convenient way to make calls to 100,000+ Machine Learning models, making it easy to incorporate AI functionality into your Supabase Edge Functions.

Setup

  • Open your Supabase project dashboard or create a new project.
  • Create a new bucket called images.
  • Generate TypeScript types from remote Database.
  • Create a new Database table called image_caption.
    • Create id column of type uuid which references storage.objects.id.
    • Create a caption column of type text.
  • Regenerate TypeScript types to include new image_caption table.
  • Deploy the function to Supabase: supabase functions deploy huggingface-image-captioning.
  • Create the Database Webhook in the Supabase Dashboard to trigger the huggingface-image-captioning function anytime a record is added to the storage.objects table.

Generate TypeScript types

To generate the types.ts file for the storage and public schemas, run the following command in the terminal:


_10
supabase gen types typescript --project-id=your-project-ref --schema=storage,public > supabase/functions/huggingface-image-captioning/types.ts

Code

Find the complete code on GitHub.


_49
import { serve } from 'https://deno.land/std@0.168.0/http/server.ts'
_49
import { HfInference } from 'https://esm.sh/@huggingface/inference@2.3.2'
_49
import { createClient } from 'jsr:@supabase/supabase-js@2'
_49
import { Database } from './types.ts'
_49
_49
console.log('Hello from `huggingface-image-captioning` function!')
_49
_49
const hf = new HfInference(Deno.env.get('HUGGINGFACE_ACCESS_TOKEN'))
_49
_49
type SoRecord = Database['storage']['Tables']['objects']['Row']
_49
interface WebhookPayload {
_49
type: 'INSERT' | 'UPDATE' | 'DELETE'
_49
table: string
_49
record: SoRecord
_49
schema: 'public'
_49
old_record: null | SoRecord
_49
}
_49
_49
serve(async (req) => {
_49
const payload: WebhookPayload = await req.json()
_49
const soRecord = payload.record
_49
const supabaseAdminClient = createClient<Database>(
_49
// Supabase API URL - env var exported by default when deployed.
_49
Deno.env.get('SUPABASE_URL') ?? '',
_49
// Supabase API SERVICE ROLE KEY - env var exported by default when deployed.
_49
Deno.env.get('SUPABASE_SERVICE_ROLE_KEY') ?? ''
_49
)
_49
_49
// Construct image url from storage
_49
const { data, error } = await supabaseAdminClient.storage
_49
.from(soRecord.bucket_id!)
_49
.createSignedUrl(soRecord.path_tokens!.join('/'), 60)
_49
if (error) throw error
_49
const { signedUrl } = data
_49
_49
// Run image captioning with Huggingface
_49
const imgDesc = await hf.imageToText({
_49
data: await (await fetch(signedUrl)).blob(),
_49
model: 'nlpconnect/vit-gpt2-image-captioning',
_49
})
_49
_49
// Store image caption in Database table
_49
await supabaseAdminClient
_49
.from('image_caption')
_49
.insert({ id: soRecord.id!, caption: imgDesc.generated_text })
_49
.throwOnError()
_49
_49
return new Response('ok')
_49
})