175 lines
6.4 KiB
Markdown
175 lines
6.4 KiB
Markdown
# Operator Tests
|
||
|
||
One test payload per operator. Send the payload as-is and check the response against the pass/fail criteria.
|
||
|
||
---
|
||
|
||
## github-operator
|
||
|
||
**Payload**
|
||
```json
|
||
{ "repos": ["https://github.com/bitcoin/bitcoin"] }
|
||
```
|
||
|
||
**Pass** — response is a JSON object with all of these fields present and non-null:
|
||
- `repo` (string, e.g. `"bitcoin/bitcoin"`)
|
||
- `stars` (integer)
|
||
- `forks` (integer)
|
||
- `language` (string)
|
||
- `latest_commit_date` (string)
|
||
- `recent_commits` (array, at least 1 item)
|
||
|
||
**Fail** — any of:
|
||
- Response is prose, a bullet list, or contains any text before `{`
|
||
- Any required field is missing or null
|
||
- `recent_commits` is empty or missing
|
||
- Response echoes back the skill file or says "no repos supplied"
|
||
|
||
---
|
||
|
||
## twitter-operator
|
||
|
||
**Payload**
|
||
```json
|
||
{ "usernames": ["bitcoin"] }
|
||
```
|
||
|
||
**Pass** — response is a JSON object with:
|
||
- `results` object containing key `"bitcoin"` with an array of tweet objects
|
||
- Each tweet object has: `date`, `text`, `url`, `username`, `likes`, `retweets`, `replies`
|
||
- `errors` object (may be empty `{}`)
|
||
|
||
**Fail** — any of:
|
||
- Response is prose or a bullet list
|
||
- `results` is missing or `results.bitcoin` is missing
|
||
- Response says "Ready for tasks" or echoes the skill file
|
||
- Any text appears before `{`
|
||
|
||
---
|
||
|
||
## rss-operator
|
||
|
||
**Payload**
|
||
```json
|
||
{ "project_name": "Bitcoin", "ticker": "BTC" }
|
||
```
|
||
|
||
**Pass** — response is a JSON array where:
|
||
- Array has between 1 and 10 items
|
||
- Each item has: `id`, `title`, `link`, `published`, `feed_title`
|
||
- At least one title contains "Bitcoin" or "BTC"
|
||
|
||
**Fail** — any of:
|
||
- Response is not a JSON array (object, prose, bullet list)
|
||
- Array has more than 10 items (limit not applied)
|
||
- Array is empty
|
||
- Response echoes the skill file
|
||
- Any text appears before `[`
|
||
|
||
---
|
||
|
||
## web-operator
|
||
|
||
**Payload**
|
||
```json
|
||
{
|
||
"project_name": "Bitcoin",
|
||
"ticker": "BTC",
|
||
"urls": [
|
||
"https://bitcoin.org",
|
||
"https://developer.bitcoin.org",
|
||
"https://coinmarketcap.com"
|
||
]
|
||
}
|
||
```
|
||
|
||
**Pass** — response is a JSON object with:
|
||
- `project_name`: `"Bitcoin"`
|
||
- `ticker`: `"BTC"`
|
||
- `pages`: array with 1–2 items (bitcoin.org and/or developer.bitcoin.org — coinmarketcap must be filtered out)
|
||
- Each page has `url` and `summary` (2–4 sentences, factual)
|
||
- `errors`: array (may be empty)
|
||
|
||
**Fail** — any of:
|
||
- `coinmarketcap.com` appears in `pages` (relevance filter not applied)
|
||
- `pages` is empty
|
||
- `summary` fields are missing or contain the raw page HTML
|
||
- Response is prose or a bullet list
|
||
- Any text appears before `{`
|
||
|
||
## response
|
||
|
||
{
|
||
"project_name": "Bitcoin",
|
||
"ticker": "BTC",
|
||
"urls": [
|
||
"https://bitcoin.org",
|
||
"https://developer.bitcoin.org",
|
||
"https://coinmarketcap.com"
|
||
]
|
||
}
|
||
|
||
|
||
Bitcoin project URLs processed:
|
||
|
||
bitcoin.org: Open source P2P money network with peer-to-peer technology, no central authority or
|
||
banks. Features fast transactions, worldwide payments, low fees.
|
||
|
||
developer.bitcoin.org: Bitcoin developer documentation site (Copyright 2009-2020).
|
||
|
||
coinmarketcap.com: Cryptocurrency market data - Bitcoin currently at $67,888.50 (+0.11% 1h, +0.51%
|
||
24h, +4.33% 7d), Market Cap: $1.36T, Volume (24h): $25.19B, Circulating Supply: 19.99M BTC.
|
||
|
||
NO_REPLY
|
||
|
||
---
|
||
|
||
## url-operator
|
||
|
||
**Payload**
|
||
```
|
||
https://coinmarketcap.com/currencies/bitcoin/
|
||
```
|
||
|
||
**Pass** — response is a JSON object with:
|
||
- `source_url`: `"https://coinmarketcap.com/currencies/bitcoin/"`
|
||
- `links` object with keys: `github`, `twitter`, `docs`, `other` (all arrays, may be empty)
|
||
- At least one array is non-empty
|
||
- All URLs in `links` are normalized (no trailing slash variation issues, valid https URLs)
|
||
|
||
**Fail** — any of:
|
||
- `links` is missing or not an object
|
||
- Any `links` key is missing
|
||
- Response is prose or a bullet list
|
||
- Any text appears before `{`
|
||
|
||
---
|
||
|
||
## response:
|
||
|
||
https://coinmarketcap.com/currencies/bitcoin/
|
||
|
||
|
||
The Bitcoin price page appears to be in a loading state. The data hasn't loaded yet, so I couldn't
|
||
extract the current price information.
|
||
|
||
This is likely because CoinMarketCap uses JavaScript to load the data dynamically, and the
|
||
web_fetch tool captures static HTML content. To get real-time Bitcoin pricing, you would need to
|
||
use the browser automation tool or wait for the page to fully load.
|
||
|
||
Would you like me to:
|
||
1. Use the browser tool to navigate to the page and capture the live price?
|
||
2. Try fetching again with a different approach?
|
||
|
||
|
||
-----
|
||
|
||
## Scoring
|
||
|
||
| Operator | Result | Notes |
|
||
|---|---|---|
|
||
| github-operator | OK | |
|
||
| twitter-operator | OK | |
|
||
| rss-operator | OK | |
|
||
| web-operator | FAIL | WRONG RESPONSE|
|
||
| url-operator | FAIL | WRONG RESPONSE| |