Tom Koole

The new year.

So schools been going well enough this term to give me some free time so I decided that I'd write a little charitable app to spread awareness for Mental Health. As Bell is offering to give 5 cents for every SMS or long distance call toward their Mental Health Fund on Feb. 8th, 2012 I've created a simple command-response server app that sends out a greeting via SMS and waits for commands eventually providing more information about the fundraiser and all the while increasing the number of texts sent that day. For more information about Let's talk visit bells website http://letstalk.bell.ca/ My app can be found on the Android Market here https://market.android.com/details?id=com.tomkoole.letstalk' And the source for the app can be found on github here https://github.com/kewleus/letstalk

Holiday Hours

As I have more time over the holidays I've been working on some more android projects. An IR-Remote for controlling my home theatre setup from my phone. IR-Remote An Android Audio Player from streaming live audio from a computer to your phone. Android-Audio-Player And of course I'm still working on the Roll up the Rim Tracker project. RUTRT I've also been working on Project Euler as part of the Brock CSC group. project-euler on git

End of 2011

With 2011 coming to a close and exams in full gear I decided to take a break from studying and update my mobile app for Roll Up The Rim to get ready for February. You can now install and register from the android app or on the website. You can browse the android market here. Obviously result collection won't begin until the contest actually opens. If you're here looking for Ogre or FFmpeg work see the following links. Ogre 3D & Gorilla Tutorial FFmpeg decoding in C++ ADDENDUM: So I've started a new opensource project over on github for developing an ir remote on the arduino. It includes projects for the server, arduino and android phone as the remote. Check it out below. IR-Remote

FFmpeg C++ Wrapper for Audio/Video Processing

This code is an example of how to use FFmpeg in C++ to decode nearly any audio / video file for use in processing. There is no timing code so you will need to figure that out on your own if you want to use it to create a player. To create most of this code I followed dranger's tutorial for a player in C and converted it over to C++.

I don't particularily feel like writing a tutorial though the organization is fairly straight forward and most of the internal code is built on dranger's work, if you want a better explanation send me an email at tom@tomkoole.com

The FFmpeg wrapper contains 3 objects and a helper class:

KFWrap: the primary object for opening the file, pulling data and cleaning up.
KFAudioStream: the object for decoding/buffering audio samples
KFVideoStream: the object for decoding/buffering video as well as conversion from YUV to RGB
KFPackets: handles the raw data buffering

It also contains several helper headers for windows / linux support. I haven't tried to compile it on linux however I don't think there's anything that would cause it to fail.

I am using a fairly recent version of FFmpeg from zeranoe(git-39dbe9b) and Boost 1.46.1 for threading purposes, for windows you can get precompiled binaries from Boost Pro. If you are building with Visual Studio you will probably need to grab the C99 inttype headers from Google Code (they are included in the zip).

This is by no means a complete work and may be updated at any time. You can download the source here or copy it from below.

KFExport: used to export functions to the lib for compiling a dll on windows. (kfexport.h)


#pragma once
#ifndef KFEXPORT_H
#define KFEXPORT_H

#ifdef _WIN32
#define DllImport __declspec( dllimport )
#define DllExport __declspec( dllexport )
#elif 
#define DllImport EMPTY
#define DllExport EMPTY
#endif //_WIN32

#endif //KFEXPORT_H

KFSleep: used to have a universal sleep command independant of compiler.(kfsleep.h)


#pragma once
#ifndef KFSLEEP_H
#define KFSLEEP_H

#ifdef _WIN32
	#include <windows.h>
	#define SLEEP(t) Sleep(t)
#else
	#include <unistd.h>
	#define SLEEP(t) sleep(t);
#endif //_WIN32

#endif //KFSLEEP_H

KFHeader: a common include file for all objects (kfheader.h)


#pragma once
#ifndef KFHEADER_H
#define KFHEADER_H

#include <stdio.h>
#include <string>
#include <boost/cstdint.hpp>
#include <boost/thread/thread.hpp>
#include <boost/thread/mutex.hpp>
#include <boost/thread/condition.hpp>

extern "C" 
{
	#include <avcodec.h>
	#include <avformat.h>
	#include <swscale.h>
}

#include "kfsleep.h"
#include "kfexport.h"

#endif//KFHEADER_H

kfwrap.h


#pragma once
#ifndef KFWRAP_H
#define KFWRAP_H

#include "kfheader.h"
#include "kfastream.h"
#include "kfvstream.h"

// Windows or C++ doesn't like the original
// value of this define so I just redefined it.
#ifdef AV_NOPTS_VALUE
#undef AV_NOPTS_VALUE
#endif
#define AV_NOPTS_VALUE 0x8000000000000000

#define MAX_MEMORY 1000000000

// Decode Thread State
enum BufferState {
	BUFFER_STATE_STOPPED = 0,
	BUFFER_STATE_RUNNING,
	BUFFER_STATE_PAUSED
};

class KFWrap
{
public:
	DllExport KFWrap(std::string file);
	DllExport ~KFWrap();

	DllExport bool				isLoaded() { return (mFormatCtx != nullptr); };

	DllExport KFVideoStream*	getVideoStream();
	DllExport KFAudioStream*	getAudioStream(unsigned short stream);
	DllExport int				getNumberOfAudioStreams();
	DllExport void				discardVideo(bool yes);
	DllExport void				discardAudio(bool yes);

	DllExport void				freeSample(KFAudioSample sample);
	DllExport void				freeSample(KFVideoFrameRGB frame);
	DllExport void				freeSample(KFVideoFrameYUV frame);
		
	// -- Buffering Controls --
	DllExport void				startBuffering();
	DllExport void				pauseBuffering();
	DllExport void				stopBuffering();

private:
	void						getMoreData();

	PacketQueue*				getAudioPacketQueue(unsigned short streamid);

private:
	std::string					mFileName;
	AVFormatContext				*mFormatCtx;

	std::vector<KFAudioStream*> mAudioStreams;
	KFVideoStream*				mVideoStream;
	
	BufferState					mBufferState;
	boost::thread				mBufferThread;

	bool						mDiscardVideo;
	bool						mDiscardAudio;
};

#endif//KFWRAP_H

kfwrap.cpp


#include "kfwrap.h"

KFWrap::KFWrap(std::string file)
{
	if(file.size() == 0) 
		return;
	this->mFileName = file;
	mVideoStream = 0;
	av_register_all();

	av_log_set_level(0);

	if(av_open_input_file(&mFormatCtx, mFileName.c_str(), NULL, 0, NULL)) 
		return;

	if(av_find_stream_info(mFormatCtx) < 0)
		return;

	for(unsigned short i = 0; i < mFormatCtx->nb_streams; i++)
		if(mFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO)
			mVideoStream = new KFVideoStream(mFormatCtx, i);
		else if(mFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_AUDIO)
			mAudioStreams.push_back(new KFAudioStream(mFormatCtx, i));

	mBufferState = BUFFER_STATE_STOPPED;

	mDiscardVideo = false;
	mDiscardAudio = false;
}

KFWrap::~KFWrap() 
{
	if(mBufferState != BUFFER_STATE_STOPPED)
	{
		mBufferState = BUFFER_STATE_STOPPED;
		boost::system_time const timeout = boost::get_system_time() + boost::posix_time::milliseconds(500);
		mBufferThread.timed_join(timeout);
	}

	if(!mDiscardVideo && mVideoStream)
		mVideoStream->Close();
	mVideoStream = 0;
	
	if(!mDiscardAudio) 
	{
		while(mAudioStreams.size() > 0) {
			KFAudioStream* stream = mAudioStreams.back();
			mAudioStreams.pop_back();
			stream->Close();
			delete stream;
		}
	}

	if(mFormatCtx)
		av_close_input_file(mFormatCtx);
	mFormatCtx = 0;
}

void KFWrap::discardVideo(bool yes)
{
	mDiscardVideo = yes;
}
void KFWrap::discardAudio(bool yes)
{
	mDiscardAudio = yes;
}

KFVideoStream* KFWrap::getVideoStream()
{
	return mVideoStream;
}

KFAudioStream* KFWrap::getAudioStream(unsigned short stream)
{
	if(getNumberOfAudioStreams())
		return mAudioStreams[stream];
	else
		return 0;
}

int KFWrap::getNumberOfAudioStreams()
{
	return mAudioStreams.size();
}

void KFWrap::freeSample(KFAudioSample sample)
{
	if(sample.size)
		free(sample.data);
}

void KFWrap::freeSample(KFVideoFrameRGB frame)
{
	if(frame.allocated)
		free(frame.data);
}
void KFWrap::freeSample(KFVideoFrameYUV frame)
{
	if(frame.allocated)
		av_free(frame.frame);
}

void KFWrap::startBuffering()
{
	if(mBufferState == BUFFER_STATE_STOPPED)
	{
		mBufferState = BUFFER_STATE_RUNNING;
		mBufferThread = boost::thread(boost::bind(&KFWrap::getMoreData, this));
	}
	else
		mBufferState = BUFFER_STATE_RUNNING;
}

void KFWrap::pauseBuffering()
{
	mBufferState = BUFFER_STATE_PAUSED;
}

void KFWrap::stopBuffering()
{
	mBufferState = BUFFER_STATE_STOPPED;
	mBufferThread.join();
}

void KFWrap::getMoreData()
{
	AVPacket pkt, *packet = &pkt;
	if(mBufferState != BUFFER_STATE_STOPPED &&
		mBufferState != BUFFER_STATE_PAUSED &&
		mBufferState != BUFFER_STATE_RUNNING)
		return;

	while(this->mBufferState != BUFFER_STATE_STOPPED)
	{
		if(this->mBufferState == BUFFER_STATE_PAUSED)
		{
			SLEEP(1000);
			continue;
		}
		
		if(mFormatCtx)
		{
			if(av_read_frame(mFormatCtx, packet) < 0)
			{
				if(url_ferror(mFormatCtx->pb) == 0)
				{
					SLEEP(100);
					if(mBufferState != BUFFER_STATE_STOPPED ||
						mBufferState != BUFFER_STATE_PAUSED ||
						mBufferState != BUFFER_STATE_RUNNING)
						break;
					continue;
				}
				else
				{
					mBufferState = BUFFER_STATE_STOPPED;
					break;
				}
			}
			else
			{
				bool used = false;
				if(mVideoStream && !mDiscardVideo && packet->stream_index == mVideoStream->getStreamID())
				{
					while(mBufferState == BUFFER_STATE_RUNNING && mVideoStream->getPacketQueue()->memSize > MAX_MEMORY)
						SLEEP(10);
					KFPackets::packet_queue_put(mVideoStream->getPacketQueue(), packet);
					used = true;
				}
				else
				{
					if(!mDiscardAudio)
						for(int i = 0; i < mAudioStreams.size(); i ++)
							if(packet->stream_index == mAudioStreams[i]->getStreamID())
							{
								while(mBufferState == BUFFER_STATE_RUNNING && mAudioStreams[i]->getPacketQueue()->memSize > MAX_MEMORY)
									SLEEP(10);
								KFPackets::packet_queue_put(mAudioStreams[i]->getPacketQueue(), packet);
								used = true;
								break;
							}
				}
				if(!used)
				{
					av_free_packet(packet);
				}
			}
		}
	}
}

kfastream.h


#pragma once
#ifndef KFASTREAM_H
#define KFASTREAM_H

#include "kfheader.h"
#include "kfexport.h"

// Audio Sample for exporting data
typedef struct KFAudioSample
{
	uint8_t			*data;
	AVSampleFormat	bitfmt;
	int				bitStride;
	int				size;
} KFAudioSample;

class KFAudioStream
{
public:
	KFAudioStream(AVFormatContext* pFormatCtx, int stream);
	PacketQueue*			getPacketQueue()	{ return mPQ; };
	void					Close();
	
	DllExport int			getNumberOfChannels();
	DllExport int			getStreamID()		{ return mStreamID; };
	DllExport unsigned int	getSampleRate()		{ return mSampleRate; };
	DllExport long			getDuration()		{ return mDuration; };


	DllExport KFAudioSample getNextSample(unsigned int length);
	DllExport KFAudioSample getSampleChannel(KFAudioSample sample, unsigned short channel);
private:
	int decodeFrame(double* pts);

private:
	AVFormatContext				*mFormatCtx;
	AVCodecContext				*mCodecCtx;
	AVCodec						*mCodec;
	AVStream					*mStream;
	int							mStreamID;

	long						mDuration;
	unsigned int				mSampleRate;
	AVSampleFormat				mSampleFormat;
	int							mSampleBitStride;

	PacketQueue*				mPQ;
	Packet						mPacket;

	DECLARE_ALIGNED(16, uint8_t, mBuffer[AVCODEC_MAX_AUDIO_FRAME_SIZE*(3/2)]);
	unsigned int				mBufferSize;
	unsigned int				mBufferIndex;

	unsigned int				mSubsequentFails;

	double						mClock;
};

#endif//KFASTREAM_H

kfastream.cpp


#include "kfastream.h"

KFAudioStream::KFAudioStream(AVFormatContext *pFormatCtx, int stream)
{
	mFormatCtx = pFormatCtx;
	mStreamID = stream;
	mSubsequentFails = 0;

	if(mStreamID < 0 || mStreamID >= mFormatCtx->nb_streams)
		return;
	mCodecCtx = mFormatCtx->streams[mStreamID]->codec;

	if(mCodecCtx->codec_type != AVMEDIA_TYPE_AUDIO)
		return;

	mCodec = avcodec_find_decoder(mCodecCtx->codec_id);
	if(!mCodec || (avcodec_open(mCodecCtx, mCodec) < 0)) 
		return;

	long duration = mFormatCtx->duration;
	mDuration = duration / AV_TIME_BASE;

	mSampleRate = mCodecCtx->sample_rate;
	mSampleFormat = mCodecCtx->sample_fmt;
	switch(mSampleFormat)
	{
	case AV_SAMPLE_FMT_U8:
		mSampleBitStride = 1;
		break;
	case AV_SAMPLE_FMT_S16:
		mSampleBitStride = 2;
		break;
	case AV_SAMPLE_FMT_S32:
		mSampleBitStride = 4;
		break;
	}

	mStream = mFormatCtx->streams[mStreamID];
	mBufferSize = 0;
	mBufferIndex = 0;

	mPacket = Packet();
	memset(&mPacket, 0, sizeof(mPacket));
	KFPackets::packet_queue_init(&mPQ);
}

void KFAudioStream::Close()
{
	KFPackets::packet_queue_flush(mPQ);
	delete mPQ;
}

int KFAudioStream::getNumberOfChannels()
{
	return mFormatCtx->streams[mStreamID]->codec->channels;
}

KFAudioSample KFAudioStream::getNextSample(unsigned int length)
{
	KFAudioSample samp;
	samp.size = 0;
	samp.data = 0;
	samp.bitfmt = mSampleFormat;
	samp.bitStride = mSampleBitStride;

	if(this->mStreamID < 0) return samp;

	int lb = 0;
	int length2, audio_size;
	double pts;

	//samp.data = (uint8_t*)malloc(length * samp.bitStride);
	uint8_t* pdatastart = samp.data;
	length = 1;
	while(length > 0)
	{
		if(mBufferIndex >= mBufferSize)
		{
			audio_size = decodeFrame(&pts);
			if(audio_size < 0)
			{
				free(samp.data);
				samp.data = 0;
				samp.size = 0;
				return samp;
			}
			else
			{
				mBufferSize = audio_size;
				samp.data = (uint8_t*)malloc(audio_size);
				pdatastart = samp.data;
				length = audio_size;
			}
			mBufferIndex = 0;
		}

		length2 = mBufferSize - mBufferIndex;
		if(length2 > length) length2 = length;
		memcpy(samp.data, (uint8_t*)mBuffer + mBufferIndex, length2);
		length			-= length2;
		lb				+= length2;
		samp.data		+= length2;
		mBufferIndex	+= length2;
 	}

	samp.data = pdatastart;
	samp.size = lb / samp.bitStride;
	return samp;
}

KFAudioSample KFAudioStream::getSampleChannel(KFAudioSample sample, unsigned short channel)
{
	int nChannels = getNumberOfChannels();
	
	KFAudioSample samp;
	samp.size = sample.size / nChannels;
	samp.data = 0;
	samp.bitfmt = sample.bitfmt;
	samp.bitStride = sample.bitStride;

	if(this->mStreamID < 0) return samp;
	samp.data = (uint8_t*)malloc(samp.size * samp.bitStride);

	if(channel >= nChannels)
	{
		samp.size = 0;
		samp.data = (uint8_t*)"Channel is too large";
		return samp;
	}

	for(int i = 0; i < samp.size * samp.bitStride; i += samp.bitStride)
		for(int s = 0; s < samp.bitStride; s++)
			samp.data[i + s] = sample.data[i * nChannels + channel * samp.bitStride + s];

	return samp;
}

int KFAudioStream::decodeFrame(double *pts)
{
	int length = 0, data_size = 0, n = 0;
	AVPacket* packet = &mPacket.packet;
	double pts2;
	
	mSubsequentFails = 0;
	for(;;)
	{
		while(mPacket.size > 0)
		{
			data_size = sizeof(mBuffer);
			length = avcodec_decode_audio3(mStream->codec, (int16_t*)mBuffer, &data_size, packet);
			if(length < 0)
			{
				mPacket.size = 0;
				break;
			}

			mPacket.data += length;
			mPacket.size -= length;

			if(data_size <= 0)
			{
				continue;
			}

			pts2 = mClock;
			*pts = pts2;
			n = 2 * mStream->codec->channels;
			mClock += (double)data_size / (double)(n * mStream->codec->sample_rate);

			return data_size;
		}

		if(packet->data)
			av_free_packet(packet);

		if(KFPackets::packet_queue_get(mPQ, packet, 1) < 0)
			return -1;

		if(packet->data == KFPackets::FLUSH_PACKET.data)
		{
			avcodec_flush_buffers(mStream->codec);
			continue;
		}

		mPacket.data = packet->data;
		mPacket.size = packet->size;

		if(packet->pts != AV_NOPTS_VALUE)
		{
			mClock = av_q2d(mStream->time_base) * packet->pts;
		}
	}
}

kfvstream.h


#pragma once
#ifndef KFVSTREAM_H
#define KFVSTREAM_H

#include "kfheader.h"
#include "kfexport.h"

typedef struct KFVideoFrameYUV
{
	AVFrame* frame;
	int width, height;
	bool allocated;
	KFVideoFrameYUV() 
	{
		width = 0;
		height = 0;
		frame = 0;
		allocated = false;
	}
} KFVideoFrameYUV;

typedef struct KFVideoFrameRGB
{
	uint8_t *data;
	int	 width, height;
	bool allocated;
	KFVideoFrameRGB() 
	{
		data = 0;
		width = 0;
		height = 0;
		allocated = false;
	}
} KFVideoFrameRGB;

class KFVideoStream
{
public:
	KFVideoStream(AVFormatContext* pFormatCtx, int stream);

	PacketQueue* getPacketQueue() { return mPQ; };
	int			 getStreamID() { return mStreamID; };
	void		 Close();
	
	DllExport float				getFPS();
	DllExport float				getTimeBase();
	DllExport double			getDurationSeconds();
	DllExport long long			getFrameCount();
	DllExport KFVideoFrameRGB	getNextFrame();
	DllExport KFVideoFrameYUV	getNextYUVFrame();
	DllExport KFVideoFrameRGB	getRGBFromYUV(KFVideoFrameYUV frame);

private:
	void		allocPicture();
	bool		queuePicture(AVFrame* frame, double pts);
	// video decoding
	void		getMoreFrames();

private:
	AVFormatContext					*mFormatCtx;
	AVCodecContext					*mCodecCtx;
	AVCodec							*mCodec;
	AVStream						*mStream;
	int								mStreamID;

	PacketQueue						*mPQ;

	std::vector<KFVideoFrameYUV>	mBuffer;
	unsigned int					mBufferSize;
	unsigned int					mBufferWriteIndex;
	unsigned int					mBufferReadIndex;
	unsigned int					mBufferMaxLength;
	uint64_t						mBufferPts;

	double							mClock;
	struct SwsContext				*mIMGContext;

	bool							mDecoding;
	boost::thread					mDecodeThread;
	boost::mutex					mDecodeMutex;
	boost::condition				mDecodeCondition;
};

#endif//KFVSTREAM_H

kfvstream.cpp


#include "kfvstream.h"

KFVideoStream::KFVideoStream(AVFormatContext *pFormatCtx, int stream)
{
	mFormatCtx = pFormatCtx;
	mStreamID = stream;

	if(mStreamID < 0 || mStreamID >= mFormatCtx->nb_streams)
		return;

	mCodecCtx = mFormatCtx->streams[mStreamID]->codec;
	
	if(mCodecCtx->codec_type != AVMEDIA_TYPE_VIDEO)
		return;

	mCodec = avcodec_find_decoder(mCodecCtx->codec_id);
	if(!mCodec || (avcodec_open(mCodecCtx, mCodec) < 0))
		return;

	mStream = mFormatCtx->streams[mStreamID];
	mBufferReadIndex = 0;
	mBufferWriteIndex = 0;
	mBufferMaxLength = 5;
	mBufferSize = 0;
	for(int i = 0; i < mBufferMaxLength; i++)
	{
		mBuffer.push_back(KFVideoFrameYUV());
		mBuffer[i].height = mCodecCtx->height;
		mBuffer[i].width = mCodecCtx->width;
		mBuffer[i].frame = 0;
		mBuffer[i].allocated = false;
	}
	KFPackets::packet_queue_init(&mPQ);
	
	mIMGContext = sws_getContext(mCodecCtx->width, mCodecCtx->height, mCodecCtx->pix_fmt, mCodecCtx->width, mCodecCtx->height, PIX_FMT_BGR24, SWS_BICUBIC, NULL, NULL, NULL);
	
	mDecoding = true;
	mDecodeThread = boost::thread(boost::bind(&KFVideoStream::getMoreFrames, this));
}

void KFVideoStream::Close()
{
	av_free(mIMGContext);
	KFPackets::packet_queue_flush(mPQ);
	if(mDecoding)
	{
		mDecoding = false;
		boost::system_time const timeout = boost::get_system_time() + boost::posix_time::milliseconds(500);
		mDecodeThread.timed_join(timeout);
	}
	delete mPQ;
}

float KFVideoStream::getFPS()
{
	if(mFormatCtx)
	{
		AVRational fps = mStream->avg_frame_rate;
		return (float)fps.num / (float)fps.den;
	}
	else
		return -1;
}
float KFVideoStream::getTimeBase()
{
	if(mFormatCtx)
	{
		return mStream->time_base.num / mStream->time_base.den;
	}
	else
		return -1;
}

double KFVideoStream::getDurationSeconds() 
{
	if(mFormatCtx)
	{
		return (double)mFormatCtx->duration / (double)AV_TIME_BASE;
	}
	else
		return -1;
}

long long KFVideoStream::getFrameCount()
{
	if(mFormatCtx)
	{
		return mStream->nb_frames;
	}
	else
		return -1;
}

KFVideoFrameYUV KFVideoStream::getNextYUVFrame()
{
	KFVideoFrameYUV vFrame;
	vFrame.height = this->mStream->codec->height;
	vFrame.width = this->mStream->codec->width;
	vFrame.frame = 0;
	vFrame.allocated = false;

	if(this->mStream)
	{
		if(this->mBufferSize == 0)
		{
			vFrame.frame = 0;
			vFrame.height = 0;
			vFrame.width = 0;
			return vFrame;
		}
		else if(this->mBuffer[this->mBufferReadIndex].frame != nullptr)
		{
			KFPackets::ScopedLock lock(this->mDecodeMutex);

			AVFrame* frame = mBuffer[mBufferReadIndex].frame;
			vFrame.frame = frame;

			mBuffer[mBufferReadIndex].frame = 0;
			mBuffer[mBufferReadIndex].allocated = false;

			--this->mBufferSize;
			this->mDecodeCondition.notify_one();

			vFrame.allocated = true;
			if(++this->mBufferReadIndex >= this->mBufferMaxLength)
				this->mBufferReadIndex = 0;
		}
		else
		{
			vFrame.frame = 0;
			vFrame.height = 0;
			vFrame.width = 0;
			return vFrame;
		}
	}
	else
	{
		vFrame.frame = 0;
		vFrame.height = 0;
		vFrame.width = 0;
		return vFrame;
	}
	return vFrame;
}

KFVideoFrameRGB KFVideoStream::getRGBFromYUV(KFVideoFrameYUV frame)
{
	KFVideoFrameRGB vFrame;
	vFrame.height = this->mStream->codec->height;
	vFrame.width = this->mStream->codec->width;
	vFrame.data = (uint8_t*)malloc(avpicture_get_size(PIX_FMT_BGR24, vFrame.width, vFrame.height));
	vFrame.allocated = false;
	
	if(this->mStream)
	{
		AVFrame* yframe = frame.frame;
		AVFrame* pFrameRGB = avcodec_alloc_frame();
		if(pFrameRGB == nullptr) 
		{
			av_free(pFrameRGB);
			free(vFrame.data);
			vFrame.data = 0;
			vFrame.height = 0;
			vFrame.width = 0;
			return vFrame;
		}

		avpicture_fill((AVPicture*)pFrameRGB, (uint8_t*)vFrame.data, PIX_FMT_BGR24, mCodecCtx->width, mCodecCtx->height);
		sws_scale(mIMGContext, yframe->data, yframe->linesize, 0, mCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);	
		vFrame.allocated = true;
	}
	else
	{
		free(vFrame.data);
		vFrame.data = 0;
		vFrame.height = 0;
		vFrame.width = 0;
		return vFrame;
	}
	return vFrame;
}

KFVideoFrameRGB KFVideoStream::getNextFrame()
{
	KFVideoFrameRGB vFrame;
	vFrame.height = this->mStream->codec->height;
	vFrame.width = this->mStream->codec->width;
	vFrame.data = (uint8_t*)malloc(avpicture_get_size(PIX_FMT_BGR24, vFrame.width, vFrame.height));
	vFrame.allocated = false;

	if(this->mStream)
	{
		if(this->mBufferSize == 0)
		{
			free(vFrame.data);
			vFrame.data = 0;
			vFrame.height = 0;
			vFrame.width = 0;
			return vFrame;
		}
		else if(this->mBuffer[this->mBufferReadIndex].frame != nullptr)
		{
			AVFrame* frame = mBuffer[mBufferReadIndex].frame;
			AVFrame* pFrameRGB = avcodec_alloc_frame();
			if(pFrameRGB == nullptr) 
			{
				av_free(pFrameRGB);
				free(vFrame.data);
				vFrame.data = 0;
				vFrame.height = 0;
				vFrame.width = 0;
				return vFrame;
			}
			// convert ycbcr(yuv) to rgb
			avpicture_fill((AVPicture*)pFrameRGB, (uint8_t*)vFrame.data, PIX_FMT_BGR24, mCodecCtx->width, mCodecCtx->height);
			sws_scale(mIMGContext, frame->data, frame->linesize, 0, mCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
			
			KFPackets::ScopedLock lock(this->mDecodeMutex);
			av_free(mBuffer[mBufferReadIndex].frame);
			mBuffer[mBufferReadIndex].frame = 0;
			mBuffer[mBufferReadIndex].allocated = 0;

			--this->mBufferSize;
			this->mDecodeCondition.notify_one();

			vFrame.allocated = true;
			if(++this->mBufferReadIndex >= this->mBufferMaxLength)
				this->mBufferReadIndex = 0;
		}
		else
		{
			free(vFrame.data);
			vFrame.data = 0;
			vFrame.height = 0;
			vFrame.width = 0;
			return vFrame;
		}
	}
	else
	{
		free(vFrame.data);
		vFrame.data = 0;
		vFrame.height = 0;
		vFrame.width = 0;
		return vFrame;
	}
	return vFrame;
}

void KFVideoStream::getMoreFrames()
{
	double pts = 0;
	AVFrame* frame = 0;
	AVPacket pk, *packet = &pk;
	int length = 0, frameFinished = 0;
	frame = avcodec_alloc_frame();
	
	while(mDecoding)
	{
		if(KFPackets::packet_queue_get(this->mPQ, packet, 1) < 0)
			break;

		if(packet->data == KFPackets::FLUSH_PACKET.data)
		{
			avcodec_flush_buffers(this->mStream->codec);
			continue;
		}

		pts = 0;
		this->mBufferPts = packet->pts;
		length = avcodec_decode_video2(this->mCodecCtx, frame, &frameFinished, packet);

		if(packet->dts == AV_NOPTS_VALUE && frame->opaque && *(uint64_t*)frame->opaque != AV_NOPTS_VALUE)
			pts = *(uint64_t*)frame->opaque;
		else if(packet->dts != AV_NOPTS_VALUE)
			pts = packet->dts;
		else
			pts = 0;

		pts *= av_q2d(this->mStream->time_base);

		// did we get a frame
		if(frameFinished)
		{
			if(!this->queuePicture(frame, pts))
			{
				break;
			}
			else
			{
				frame = avcodec_alloc_frame();
			}
			av_free_packet(packet);
		}
	}
	av_free(frame);
}
bool KFVideoStream::queuePicture(AVFrame* frame, double pts)
{
	KFVideoFrameYUV *vFrame = 0;

	KFPackets::ScopedLock lock(this->mDecodeMutex);
	while(this->mBufferSize >= this->mBufferMaxLength && this->mDecoding)
	{
		this->mDecodeCondition.wait(lock);
	}
	lock.unlock();

	if(!mDecoding) return false;
	
	KFPackets::ScopedLock lock2(this->mDecodeMutex);
	vFrame = &this->mBuffer[this->mBufferWriteIndex];
	while(vFrame->frame && this->mDecoding)
	{
		this->mDecodeCondition.wait(lock2);
	}
	lock2.unlock();

	if(!vFrame->frame)
	{
		vFrame->frame = frame;
		vFrame->frame->pts = pts * 1000000;

		if(++this->mBufferWriteIndex == this->mBufferMaxLength)
			this->mBufferWriteIndex = 0;

		KFPackets::ScopedLock lock(this->mDecodeMutex);
		++this->mBufferSize;
	}
	return true;
}

kfpackets.h


#pragma once
#ifndef PACKETHANDLER_H
#define PACKETHANDLER_H

#include <boost/thread/thread.hpp>
#include <boost/thread/mutex.hpp>
#include <boost/thread/condition.hpp>

extern "C" 
{
#include <avcodec.h>
#include <avformat.h>
#include <swscale.h>
}

class PacketQueue 
{
public:
	AVPacketList				*FirstPacket, *LastPacket;
	int							numPackets;
	int							memSize;
	int							lockedSeconds;
	mutable boost::mutex		mutex;
	boost::condition_variable	cond;

	PacketQueue() 
	{
		FirstPacket = 0;
		LastPacket = 0;
		numPackets = 0;
		memSize = 0;
		lockedSeconds = 0;
	}
};

typedef struct Packet
{
	AVPacket			packet;
	uint8_t				*data;
	unsigned int		size;
} Packet;

class KFPackets
{
public:
	static void packet_queue_init(PacketQueue** queue);
	static void packet_queue_flush(PacketQueue* queue);
	static bool packet_queue_put(PacketQueue* queue, AVPacket* pkt);
	static int  packet_queue_get(PacketQueue* queue, AVPacket* pkt, int block);

	static AVPacket FLUSH_PACKET;
	typedef boost::mutex::scoped_lock ScopedLock;
};

#endif //PACKETHANDLER_H

kfpackets.cpp


#include "kfpackets.h"

AVPacket KFPackets::FLUSH_PACKET;

void KFPackets::packet_queue_init(PacketQueue **queue)
{
	if(FLUSH_PACKET.data != (uint8_t*)"FLUSH")
	{
		av_init_packet(&FLUSH_PACKET);
		FLUSH_PACKET.data = (uint8_t*)"FLUSH";
	}
	(*queue) = new PacketQueue();
	(*queue)->FirstPacket = 0;
	(*queue)->LastPacket = 0;
	(*queue)->lockedSeconds = 0;
	(*queue)->memSize = 0;
	(*queue)->numPackets = 0;
}

bool KFPackets::packet_queue_put(PacketQueue *queue, AVPacket *pkt)
{
	ScopedLock lock(queue->mutex);
	bool was_empty = !queue->numPackets;

	AVPacketList *packet_list = 0;
	if(pkt != &FLUSH_PACKET && av_dup_packet(pkt) < 0)
		return false;
	packet_list = (AVPacketList*)av_malloc(sizeof(AVPacketList));
	if(!packet_list)
		return false;

	packet_list->pkt  = *pkt;
	packet_list->next = nullptr;

	if(!queue->LastPacket)
		queue->FirstPacket = packet_list;
	else
		queue->LastPacket->next = packet_list;

	queue->LastPacket = packet_list;
	queue->numPackets++;
	queue->memSize += packet_list->pkt.size;
	
	lock.unlock();

	if(was_empty)
		queue->cond.notify_one();
	
	return true;
}

int KFPackets::packet_queue_get(PacketQueue *queue, AVPacket *pkt, int block)
{
	ScopedLock lock(queue->mutex);

	AVPacketList *packet_list;
	queue->lockedSeconds = 0;

	for(;;)
	{
		packet_list = queue->FirstPacket;
		if(packet_list)
		{
			queue->FirstPacket = packet_list->next;
			if(!queue->FirstPacket)
				queue->LastPacket = nullptr;
			queue->numPackets--;
			queue->memSize -= packet_list->pkt.size;
			*pkt = packet_list->pkt;
			av_free(packet_list);
			return 1;
		}
		else if (!block)
		{
			return 0;
		}
		else
		{
			if(queue->lockedSeconds++ > 1) // 2 second timeout
				return -1;
			boost::system_time tAbsoluteTime = boost::get_system_time() + boost::posix_time::milliseconds(1000);
			queue->cond.timed_wait(lock, tAbsoluteTime);
		}
	}

	return 0;
}

void KFPackets::packet_queue_flush(PacketQueue *queue)
{
	AVPacketList *packet_list1, *packet_list2;
	
	ScopedLock lock(queue->mutex);

	for(packet_list1 = queue->FirstPacket; packet_list1 != NULL; packet_list1 = packet_list2)
	{
		packet_list2 = packet_list1->next;
		av_free_packet(&packet_list1->pkt);
		av_freep(&packet_list1);
	}

	queue->LastPacket = nullptr;
	queue->FirstPacket = nullptr;
	queue->numPackets = 0;
	queue->memSize = 0;
}

As per dranger, this work is licensed under the Creative Commons Attribution-Share Alike 2.5 License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.5/ or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA. Code examples are based off of FFplay, Copyright (c) 2003 Fabrice Bellard, and a tutorial by Martin Bohme.

Gorilla Series Addendum

Part 1 - Part 2 - Part 3 - Part 4 - Part 5 - Part 6 I have gone through and removed the dependency on OgreFramework although this requires a Gui object to be past into most constructors. The Gui should be the same per Scene though I don't think it has to be, just for every Gui object you need to create a screen which increases the batch count. Another notice should be given to the order in which you create the objects as this will affect the order they are drawn. I also realized that I had not created the initial screen that is used by each object, so after initializing the Gui object call createScreen() like so. Gui* pGui = new Gui(); pGui->createScreen(pViewport, "world", "WorldState"); I have modified the tutorial to reflect these changes.

Tom Koole

The new year.

Holiday Hours

End of 2011

FFmpeg C++ Wrapper for Audio/Video Processing

Gorilla Series Addendum

Top Abstracts

Recent Blogs

Recent Abstracts

Tom Koole

The new year. (function() { var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/plusone.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();

Holiday Hours (function() { var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/plusone.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();

End of 2011 (function() { var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/plusone.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();

FFmpeg C++ Wrapper for Audio/Video Processing (function() { var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/plusone.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();

Gorilla Series Addendum (function() { var po = document.createElement('script'); po.type = 'text/javascript'; po.async = true; po.src = 'https://apis.google.com/js/plusone.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(po, s); })();

Top Abstracts

Recent Blogs

Recent Abstracts

The new year.

Holiday Hours

End of 2011

FFmpeg C++ Wrapper for Audio/Video Processing

Gorilla Series Addendum